VScript: Controllable Script Generation with Visual Presentation

1 Introduction

• Prior systems omitted scene descriptions, did not present scripts through video, and lacked control over elements such as genre.Complete scripts require chronological scene descriptions and dialogue, alongside rich and diverse content.Source: §1 · p. 1
• VScript is a controllable script-generation system designed to include all essential script components.Its framework uses hierarchical planning and follows video-making guidelines.Source: §1 · p. 1
• Users provide a genre and starting words to guide a genre-specific plot, which then guides dialogue and scene-description generation.Video retrieval presents the generated script visually; the retrieved presentation is intended as a rough draft for engagement and idea iteration.Source: Figure 1 · p. 1, §1 · p. 2, Figure 2 · p. 2
• VScript introduces plot-guided dialogue generation as inverse dialogue summarization, and the authors report that their scripts are controllable and preferred by humans.They describe VScript as the first system to tackle controllable script generation for genre and storyline.Source: §1 · p. 2

Story Generation

• Story-generation methods model event sequences, transform a premise into a passage, generate from keyword storylines or character-and-event phrases, or use genre and an image as context.The cited examples include Martin et al. (2018), Fan et al. (2018), Plan-and-Write (Yao et al., 2019), Rashkin et al. (2019), and Lovenia et al. (2022).Source: §2 · p. 2

Controllable Text Generation Model

• Controllable text-generation approaches include CTRL's domain control codes, PPLM's attribute classifiers, and GeDi's use of a class-conditional language model as a discriminator.CTRL (Keskar et al., 2019), PPLM (Dathathri et al., 2019), and GeDi (Krause et al., 2020) steer generation toward desired attributes.Source: §2 · p. 2

3 Methodology

• The methodology separates VScript into script-generation and visual-presentation modules.The script module generates plot, dialogue, and scene descriptions, while the presentation module searches a video database for relevant visualization.Source: §3 · p. 2

3.1 Script Generation

• A genre-specific plot acts as an outline that hierarchically guides dialogue and scene descriptions, with dialogue conditioned on plot and scenes on dialogue.The paper uses Crime, Sci-Fi, War, and Romance as its four classic and popular plot-generation genres.Source: §3.1 · p. 3

3.1.1 Genre-Specific Plot Generation

• Plot generation uses a genre-conditioned language model and selects, from N sampled plots, the one with the highest predicted probability for the target genre.The model is fine-tuned from GPT2-large on 42,306 movie plots; a multi-class genre classifier scores candidate plots.Source: §3.1.1 · p. 3

3.1.2 Plot-Guided Dialogue Generation

• To avoid building a plot-to-dialogue dataset, dialogue generation is framed as inverse abstractive summarization and generates a whole dialogue from its summary.The model is trained on SAMSum and DialogSum in reversed form; at inference, each plot sentence is expanded into a scene and dialogue.Source: §3.1.2 · p. 3

3.1.3 Scene Description Generation

• Scene descriptions, including location, time, and context, are generated from dialogue using a model fine-tuned on paired scene-dialogue data.The corresponding scene description and dialogue are concatenated into a scene, and multiple scenes form a script.Source: §3.1.3 · p. 3

3.2 Visual Presentation

• Visual presentation retrieves clips whose captions describe actions or events similar to the script, using plot sentences as queries and filtering candidates by detected gender and location.The database includes filtered news broadcasts, documentaries, and movie recaps; visual content is used without audio, and users can substitute a preferred video database.Source: §3.2 · pp. 3–4

4 Interactive User Interface

• Users choose among Crime, Sci-Fi, War, Romance, or Genre-Free and submit starting words to steer the script's development.The script and visual presentation appear separately in the script and video areas; users can interrupt and change their inputs at any time.Source: §4 · p. 4

5.1 Baselines

• Plot-generation baselines include GPT2-large and a CC-LM; dialogue baselines use turn-by-turn DialoGPT-large and GPT2, and GPT2_E generates scripts end-to-end without a plot.Video retrieval is compared with VideoCLIP.Source: §5.1 · pp. 4–5

5.2.1 Automatic Evaluation

• VScript controls genre more effectively than GPT2-large, with only a slight reduction in fluency.Table 1 reports the plot-generation automatic evaluation.Source: §5.2.1 · p. 5, Table 1 · p. 5
• Generating whole dialogues rather than turn-by-turn improves reference and plot similarity, while VScript dialogues and scripts show higher diversity and lower repetition than baselines.Tables 2 and 3 report automatic evaluations for plot-guided dialogue generation and scripts.Source: §5.2.1 · p. 5, Table 2 · p. 5, Table 3 · p. 5

5.2.2 Human Evaluation

• Human judges prefer VScript scripts over GPT2_E, and VScript video retrieval has slightly higher relevance than VideoCLIP.Table 4 reports human evaluation for scripts, and Table 5 reports human evaluation for video retrieval.Source: §5.2.2 · p. 5, Table 4 · p. 5, Table 5 · p. 5

6 Conclusion

• VScript is presented as the first framework for generating scripts of specific genres that follow plots.Source: §6 · p. 5
• The pipeline generates the plot first, then the script and its visual presentation, and uses inverse abstractive summarization for dialogue generation.Source: §6 · p. 6
• VScript outperforms baselines, with its effectiveness in genre control proven.The conclusion also describes the generated scripts as controllable and preferred by humans.Source: §6 · pp. 5–6

A Limitations

• Video retrieval depends partly on sentence-embedding, face-detection, and place-detection quality, while presentation also depends heavily on the video database’s quality and quantity.Source: A Limitations · p. 8
• The authors identify finer control over settings, character personalities, and event details as future work.Source: A Limitations · p. 8

B Ethical Considerations

• The collected videos are publicly available YouTube footage, but the database will not be released and the authors accept accountability for rights or terms-of-service violations.Source: B Ethical Considerations · p. 8

C Genre-Specific Plot Generation

• Predefined genre control codes guide plot generation, with each code corresponding to a genre.Source: C Genre-Specific Plot Generation · p. 8
• The class-conditional language model is trained on plots paired with their corresponding control codes.The passage also notes that a training loss is defined.Source: C Genre-Specific Plot Generation · p. 8
• The system generates 10 plots with the target genre and starting words, then selects the plot with the highest genre-classifier probability.The classifier assigns probabilities across genre classes.Source: Figure 4 · p. 8, C Genre-Specific Plot Generation · p. 8

D Video Database Construction

• The database uses YouTube news broadcasts and movie recaps; captions are sentence-tokenized and frames dominated by hosts or speakers are filtered.Source: D Video Database Construction · p. 8
• The processing pipeline detects faces and character gender, classifies scene locations, and uses a trained day/night image classifier.Source: D Video Database Construction · pp. 8–9
• An NLI-based zero-shot text classifier splits video captions by genre, and users can replace or update the database independently of video contents.Source: D Video Database Construction · p. 9

E Background Music

• Background music varies by script genre, with rapid, intense music for crime and relaxing, soothing music for romance.Source: E Background Music · p. 9

F.1 Plot Generation

• Plot generation fine-tunes GPT2-large with genre control-code prefixes and top-k sampling (k=4), alongside a BART-large genre classifier.Both models use a 3e-5 learning rate, AdamW, and WarmupDecayLR; the GPT2 baseline matches the GPT2-large settings.Source: §F.1 · p. 9

F.2 Plot-Guided Dialogue Generation

• The plot-guided dialogue model shares GPT2-large hyperparameters; DialogGPT uses a 5e-5 learning rate, while GPT2 Turn-by-Turn matches GPT2-large settings.Source: §F.2 · p. 9

F.3 Scene Description Generation

• Scene-description generation uses the GPT2-large hyperparameters specified for plot generation.Source: §F.3 · p. 9

F.4 Overall Script Generation

• The GPT2-E overall-script baseline uses the same hyperparameter set as GPT2-large in Appendix F.1.Source: §F.4 · p. 9

• Fluency is slightly higher than GPT2-E while VScript achieves script formulation.Source: G Additional Human Evaluation · p. 9
• For each model, three annotators rated 50 randomly sampled scripts; format used binary evaluation, while fluency and consistency used four-point scales.The evaluation applied individual t-tests for significance validation.Source: G Additional Human Evaluation · p. 9
• p < 0.05; VScript is considerably more consistent and logical than GPT2-E.Source: G Additional Human Evaluation · pp. 9–10