PaperlayerView exact revision on arXiv

Source-linked AI summary

SkillPE: Creativity-Oriented Cinematic Skill Evolution for Text-to-Video Prompt Engineering

Yanwei Huang, Mingxuan Zhu, Shujie Li, Shiyuan Liu, Yuanxing Zhang, Arpit Narechania

arXiv:2609.34335v1cs.CVcs.AI

TL;DR

Non-experts struggle to create cinematic text-to-video prompts, while existing approaches leave reusable cinematic structures and creative skill evolution underexplored. SkillPE evolves expert-authored cinematic skills using film references and video-based assessment. On StoryEval and VBench, it improves four-dimensional evaluation by up to 1.40 points over the strongest non-skill PE baseline and 0.51 points over seed skills.

  • Problem

    Text-to-video prompt engineering remains challenging for non-experts, and existing approaches leave reusable cinematic structures and creative skill evolution underexplored.

    Source: §1 · pp. 1–2

  • Method

    SkillPE represents expert-authored skills in fine-grained cinematic form, evolves them with resonator, dissonant, and divergent references, and assesses candidates through generated videos.

    Source: §1 · pp. 2–3

  • Results

    On a four-dimensional 7-point evaluation, SkillPE improves by up to 1.40 points over the strongest non-skill PE baseline and 0.51 points over seed skills on StoryEval and VBench.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Reusable, adaptively evolved cinematic skills are a potential alternative to purely lexical prompt rewriting, with a measurable fidelity–creativity trade-off.

    Source: §5 · p. 11

    More aggressive divergent evolution can increase creative expression while increasing the risk of deviating from user intent; assessed selection mitigates but does not eliminate this trade-off.

    Source: A Limitations and Future Work · p. 14

Abstract

from arXiv · show

Achieving high-quality, cinematic results in text-to-video generation remains challenging for non-experts, whose prompts often lack professional narrative and creative design. We propose SkillPE, a prompt engineering (PE) framework that evolves reusable cinematic skills from expert-authored seeds. SkillPE represents shot logic, composition, lighting, sound design, and other filmmaking cues in a fine-grained format, and retrieves movie references categorized as resonators (good matches), dissonants (weak matches), and divergents (creatively useful near-misses). The first two refine when and how a skill should be applied, while divergents inspire alternative cinematic realizations at different degrees of modification while preserving the user intent. Candidate skills are assessed through generated videos along prompt fidelity, cinematic quality, narrative appeal, and creativity to construct the final skill libraries. Experiments on StoryEval and VBench show improvements of up to 1.40 points over the strongest external baseline and 0.51 points over seed skills on 7-point four-dimensional evaluation, while remaining competitive on benchmark-native metrics. Overall, SkillPE offers a practical approach to balancing fidelity and creativity in cinematic text-to-video generation. Code is available at https://github.com/Ais0n/SkillPE .

1 Introduction

Text-to-video prompt engineering remains difficult for non-experts because existing approaches do not fully capture reusable cinematic structures or creative skill evolution. SkillPE builds and assesses cinematic skills, improving evaluations while balancing fidelity and creativity.

1 Introduction

  • Existing text-to-video PE methods mainly revise prompt text, while fixed agent pipelines and utility-oriented skill exploration leave reusable cinematic structures and creative breadth underexplored.This gap matters for narrative video generation, where creative breadth is described as crucial.Source: §1 · pp. 1–2
  • SkillPE combines fine-grained cinematic skill representations with reference-guided evolution and video-based assessment of candidate skills.The representations cover filmmaking cues such as shot logic, composition, lighting, and sound design.Source: §1 · pp. 2–3
  • Resonators and dissonants refine skill applicability and execution, while divergents inspire controlled alternative realizations at different mutation magnitudes.The approach aims to preserve the user’s intended semantics while exploring alternatives.Source: §1 · pp. 2–3
  • SkillPE improves evaluations on StoryEval and VBench, with gains of up to 0.51 points over seed skills and 1.40 points over the strongest non-skill PE baseline.These gains are on a four-dimensional 7-point evaluation of prompt fidelity, cinematic quality, narrative appeal, and creativity.Source: §1 · p. 3
  • The paper characterizes a fidelity–creativity trade-off in cinematic skill evolution and reports that a blinded ten-annotator study confirms gains over Seed Skill and non-skill PE baselines.Creativity increases with mutation scope while prompt fidelity decreases.Source: §1 · p. 3

2 Related Work

Prior work improves visual-generation prompts through rewriting and agentic workflows, and increasingly explores reusable skill libraries. SkillPE differs by evolving cinematic skills with reference-guided variants.

2.1 PE for Visual Generation

  • Text-to-video prompt methods such as Prompt-A-Video, VPO, and RAPO primarily rewrite or augment prompts rather than apply professional shot or story design.The related work positions SkillPE as reasoning through reusable cinematic skills instead of sentence-level rewriting.Source: §2.1 · p. 3

2.2 Multi-Modal LLM-Based Agents

  • Multi-agent PE systems decompose prompt work across specialized roles, but typically use fixed pipelines and static tool or skill pools that limit adaptation to varied intents and creative demands.SkillPE instead constructs libraries by generating and assessing reference-guided variants of reusable cinematic skills.Source: §2.2 · pp. 3–4

2.3 Skill Discovery and Self-Evolving Agents

  • Self-evolving skill-library work often learns from random rollouts or successful traces, which can bias discovery toward utility rather than the creative breadth needed for narrative video.The related work describes this limitation across skill synthesis and refinement research.Source: §2.3 · p. 4

3 Methodology

SkillPE evolves expert-authored cinematic skills into reusable guidance and candidate variants, then selects skills based on their generated-video performance. Its pipeline combines conservative reference-based refinement with controlled divergent exploration.

3 Methodology

  • SkillPE converts expert-authored seed skills into multiple candidates and selects skills based on realized video performance.The framework starts from a library of 20 expert-authored skills.Source: §3 · p. 4, §3.1 · p. 5

3.1 Skill Formulation

  • Each skill is a reusable cinematic guide, normalized into shot-level details such as duration, composition, lighting, time-ordered visual content, dialogue, and sound effects.This fine-grained format is intended to make guidance more actionable while preserving the original skill’s cinematic intent and applicability.Source: §3.1 · p. 5

3.2 Adaptive Skill Evolution

  • SkillPE retrieves cinematic references from LSMDC and Movie101, then selects 30 complementary examples per seed: 10 resonators, 15 divergents, and 5 dissonants.References are filtered across six dimensions, including shot logic, camera movement, composition, lighting and color, atmosphere, and audio logic.Source: §3.2 · p. 5
  • Resonators supply reusable patterns and application guidance, while dissonants provide negative applicability conditions and boundaries for conservative refinement.This branch preserves the seed skill’s core cinematic strategy.Source: §3.2 · p. 5
  • Divergents generate independent variants at three modification levels: Bold changes one or two dimensions, Wilder changes three or four including a structural dimension, and Extreme changes at least five with a specified structural or expressive transformation.Mutation is defined over nine major dimensions of the skill template.Source: §3.2 · p. 5
  • Candidate skills are assessed on 400 matched prompts across prompt fidelity, cinematic quality, narrative appeal, and creativity, then assembled into libraries using different score-based selection criteria.For candidates evaluated on the same prompt, generation settings and random seed are held constant; Top Overall averages all four dimensions.Source: §3.2 · p. 6

4 Experiments

Experiments compare SkillPE with prompt-engineering baselines across StoryEval and VBench, then test the human ratings, skill-evolution stages, prompt budgets, selection, and qualitative outputs. SkillPE improves cinematic, narrative, and creative scores, with results also showing fidelity trade-offs across mutation scope and prompt length.

4.1 Study Setups

  • The evaluation compares prompt-engineering baselines and skill-library variants on StoryEval and VBench, using a fixed four-dimensional rubric alongside benchmark-native scores.The development set is disjoint from benchmark evaluation sets, and final evaluators do not see the rewritten prompt, selected skill, or method identity.Source: §4.1 · pp. 6–7
  • SkillPE Top Overall achieves the highest Overall scores on StoryEval (5.94) and VBench (6.05), while SkillPE remains competitive but does not uniformly improve benchmark-native metrics.On LTX-2.5 VBench, Top Overall raises Overall from 5.37 to 5.88 and achieves the highest CQ (6.43), NA (5.44), and CR (5.49).Source: Table 1 · p. 7, §4.1 · p. 7

4.2 User Study

  • Across 40 matched prompts, all three final-library SkillPE variants significantly outperform Seed-skill PE after Holm correction (adjusted p < 0.05 each); Top Creativity achieves the highest overall score (5.97).Human evaluation used a 1–7 Likert scale across PF, CQ, NA, and CR.Source: §4.2 · p. 8, Table 2 · p. 8

4.3.1 Ablation of Skill Evolution Methods

  • Fine-grained seed formulation raises creativity by +0.56 on StoryEval and +0.20 on VBench; resonator-and-dissonant refinement gives positive mean CQ, NA, and Overall changes while largely preserving fidelity.The StoryEval overall increase is +0.17; the VBench overall difference (+0.10) is inconclusive.Source: §4.3.1 · pp. 8–9, Table 3 · p. 9
  • From Bold to Extreme, divergent mutations raise CR monotonically on both benchmarks while lowering PF, trading greater creative expressiveness for increased fidelity risk.The CR gains over Seed range from +0.34 to +1.32 on StoryEval and from +0.43 to +1.03 on VBench; all reported 95% confidence intervals exclude zero.Source: §4.3.1 · p. 9

4.3.2 Ablation of Prompt Length

  • At every tested prompt budget, skill-based methods exceed Direct LLM Rewrite in Creativity, and Top Overall has a higher Overall score.Top Overall’s Overall score rises from 5.56 at 100 words to 6.13 at 800 words, while Direct LLM Rewrite scores 5.55 at 400 words and 5.53 at 800 words.Source: §4.3.2 · pp. 9–10, Table 4 · p. 10

4.3.3 Ablation of Skill Selection

  • Assessed selection improves Top Overall’s Overall score over both the Seed library (+0.119) and the mean of three random libraries (+0.088), while Top Creativity reliably raises creativity over Seed.Top Creativity’s Overall advantage is not reliable; its Creativity gain over Seed is +0.251.Source: §4.3.3 · p. 10

4.4 Qualitative and Usage Analysis

  • Qualitative examples show final libraries intensifying shot progression, camera movement, composition, lighting, and audiovisual timing to fit requested events.Across evaluated prompts, outputs are shorter on average for Top Creativity and Top Overall than for Seed Skill, while the final libraries use skills from multiple evolution branches.Source: §4.4 · pp. 10–11

5 Conclusion

SkillPE combines fine-grained cinematic skill representations with reference-guided evolution and video-based selection. Experiments across two backbones and benchmarks show improved cinematic, narrative, and creative scores, alongside a measurable fidelity–creativity trade-off.

5 Conclusion

  • Across two video-generation backbones and two benchmarks, the resulting skill libraries improve cinematic, narrative, and creative evaluation scores while exposing a measurable fidelity–creativity trade-off.The authors present reusable, adaptively evolved cinematic skills as an alternative to purely lexical prompt rewriting.Source: §5 · p. 11
  • SkillPE combines fine-grained cinematic skill representations with reference-guided conservative refinement, controlled divergent exploration, and video-based assessed selection.Source: §5 · p. 11
  • The framework is presented as an alternative to purely lexical prompt rewriting through reusable, adaptively evolved cinematic skills.Source: §5 · p. 11

A Limitations and Future Work

The study is limited to two video-generation backbones, two benchmarks, and short-form 10-second videos, with a library initialized from 20 expert-authored skills and finite movie references. The authors also note a fidelity–creativity trade-off, computational overhead, and reliance on proprietary models for some multimodal stages.

A Limitations and Future Work

  • The evaluation covers two video-generation backbones, two benchmarks, and short-form 10-second videos; longer-form generation and broader model coverage remain unexplored.Source: A Limitations and Future Work · p. 14
  • The library starts from 20 expert-authored cinematic skills and finite movie references, which may not cover all genres, visual conventions, or creative preferences.Source: A Limitations and Future Work · p. 14
  • More aggressive divergent evolution can improve creative expression while increasing the risk of departing from user intent, and assessed selection mitigates but does not eliminate this trade-off.The authors suggest query-adaptive or user-controllable selection to adjust the balance between fidelity and creative ambition.Source: A Limitations and Future Work · p. 14
  • Library construction incurs a one-time cost for reference analysis, candidate generation, video rollouts, and assessment; inference adds prompt-processing calls and latency.These costs are amortized after library construction, and reducing overhead remains a direction for scaling.Source: A Limitations and Future Work · p. 14
  • Proprietary models such as Gemini 3.1 Pro are used for stages requiring joint video, audio, and text understanding; future evaluation will consider increasingly capable open-source multimodal models.The authors also state that they plan to release generated artifacts and provide detailed experimental configurations.Source: A Limitations and Future Work · pp. 14–15

B Copyright Statement

  • The authors report prior email approval from LSMDC authors and access approval for Movie101 through its official HuggingFace application process.Source: B Copyright Statement · p. 15

C Efficiency and Cost Analysis

This section examines efficiency and cost by detailing skill-library construction and inference overhead, the skills and pipeline components used to build and refine the library, and how evaluation prompts were sampled, assigned to skills, and balanced.

C Efficiency and Cost Analysis

  • Approximately 68.2K remote-model calls, 474 local embedding batches, and 2,000 video generations per backbone make library construction a one-time cost of about 625.6 aggregate GPU-hours.This cost is amortized over subsequent inference once the library has been constructed.Source: C Efficiency and Cost Analysis · p. 15
  • Inference uses two sequential calls for skill routing and prompt rewriting, or three with MiniMax-H3’s model-specific schema adaptation; shared video generation is excluded from SkillPE overhead.Source: C Efficiency and Cost Analysis · p. 15

D.1.1–D.1.2 Structure and Example of the Expert-Authored Skill

  • Expert-authored skills combine shot templates, application scenarios, shot and music logic, and examples; an “Atmosphere Building” example illustrates a panoramic-to-close-up-to-pull-out sequence.Its example links shot progression to emotional reactions and corresponding sound and music choices.Source: §D.1.1 · pp. 15–16, §D.1.2 · p. 16

D.1.3 Normalization of Expert-Authored Initial Skills

  • Normalization preserves each skill’s narrative method and applicability while adding fine-grained shot fields, including duration, setting, composition, lighting, visual content, dialogue, and sound effects.The representation is intended to bridge abstract cinematic principles and concrete generation directives.Source: §D.1.3 · p. 17

D.2 Details of Reference Video Preparation and Discovery

  • Reference discovery searches 60,521 Movie101 and LSMDC clips, retrieves candidates semantically, then assesses alignment and inspiration across six cinematic dimensions before selecting resonators, divergents, and dissonants.Selection takes 10 high-alignment resonators, 15 high-inspiration divergents, and five dissonants above cosine similarity T = 0.35.Source: §D.2 · p. 18

D.3 Details of R&D-Based Skill Refinement

  • For each source skill, Gemini 3.1 Pro independently generates a normalized seed, an R&D-refined candidate, and Bold, Wilder, and Extreme variants from the same seed and corresponding references.R&D uses positive examples to guide what to reinforce and hard negatives to define when the skill should not be forced, without rewriting its shot template.Source: §D.3 · p. 19

D.4.1 Operational Mutation Dimensions

  • Divergent mutations change one or two dimensions for Bold, three or four for Wilder, and at least five for Extreme while preserving requested semantic invariants and varying how the content is filmed.The nine dimensions cover shot structure, camera, viewpoint, composition, lighting, pacing, transitions, expressive devices, and audiovisual coordination.Source: §D.4.1 · p. 20

D.5 Details of Development Set Construction; D.5.1 VidProM Source and Preprocessing; D.5.2 Prompt Annotation

  • The development set matches 20 prompts to each of 20 source skills, drawn from a preprocessed VidProM pool of 829,589 prompts and annotated for narrative complexity and one of five categories.Preprocessing removes empty, short, technical-metadata, and toxicity-threshold-violating prompts; complexity is divided into four bins for sampling.Source: §D.5 · p. 20, §D.5.1 · pp. 20–21, §D.5.2 · p. 21

D.5.3 Skill Matching and Sampling

  • 13,000 prompts were sampled across category–complexity strata, routed to one of 20 expert skills, then greedily balanced to yield about 20 prompts per skill.Gemini 3.1 Pro performed routing at temperature 0.2.Source: §D.5.3 · p. 21

D.6 Details of Video-Based Skill Evaluation

This section describes how generated videos are scored against the original prompt on four independent dimensions and how skills are selected and applied during inference. It also covers the MiniMax-H3 prompt adaptation step used for the proposed approach and baselines, except in the length ablation study.

D.6 Details of Video-Based Skill Evaluation

  • Each candidate skill is tested on 20 development prompts using the eventual evaluation model, with Gemini 3.1 Pro scoring four dimensions independently on 7-point scales.Videos are 1344 × 768, 10 seconds, 24 FPS, and include an audio track.Source: §D.6 · p. 21
  • Prompt fidelity measures whether visible subjects, actions, relationships, state changes, attributes, and event ordering realize the user's intent.Evaluators judge observable evidence rather than inferring intended events or generation conditions.Source: §D.6 · pp. 21–22
  • Cinematic quality assesses how purposefully and coherently shot choices, camera movement, composition, staging, lighting, and transitions support the visible content.The rubric prioritizes purposeful coordination over quantity or intensity; a well-composed static shot can score highly.Source: §D.6 · p. 23
  • Narrative appeal evaluates whether events form an understandable, causally coherent progression with effective pacing and context-appropriate emotional engagement.The rubric considers transitions, state changes, buildup, emphasis, and resolution.Source: §D.6 · p. 24
  • Creativity rewards novel, expressive choices that meaningfully realize the prompt while preserving its intent, not complexity or deviation for its own sake.More shots, camera movement, elaborate lighting, or unusual additions do not automatically earn higher scores; choices must be appropriate and contribute to the expression.Source: §D.6 · pp. 25–26
  • If a video is corrupted, substantially unavailable, or technically impossible to evaluate, it is marked invalid with no score and low confidence.For invalid videos, evidence must describe the technical problem and the score must be null.Source: §D.6 · p. 26
  • Valid video evaluations use 1–4 short observable evidence statements, an integer score from 1 to 7, and high, medium, or low confidence.Evidence and rationale must be concise and grounded in observable content; the response must be strict JSON.Source: §D.6 · p. 26

D.7 Skill Routing and PE Agent

  • Gemini 3.1 Pro selects exactly one skill from the chosen library at temperature 0.1, then rewrites the prompt using that skill.The same model performs both steps.Source: §D.7 · p. 26
  • For MiniMax-H3, Gemini 3.1 Pro adapts prompts into the generator’s native format for both the proposed approach and baselines, except in the length ablation study.The adapter uses native fields, targets 220–420 English words, and validates format and event coverage.Source: §D.7 · pp. 26–27
  • The skill-conditioned rewrite must preserve requested content and event order while specifying visible action beats, environment, lighting, framing, and transitions.It produces an English 10-second text-to-video prompt and may adapt shot rhythm, camera, scale, and sound.Source: §D.7 · p. 27

E Additional Details of Evaluation

The evaluation appendix summarizes experimental configurations and tests whether SkillPE’s gains hold across video-generation seeds. SkillPE Top Overall remains stable across three seeds, with positive improvements over Seed Skill.

E Additional Details of Evaluation

F.1 Sensitivity to Video Generation Seeds

  • The seed-sensitivity test uses 50 StoryEval prompts, three MiniMax-H3 generation seeds, and paired seeds across four methods.Prompts, skill libraries, PE outputs, and other generation settings are held fixed; the experiment contains 600 videos.Source: §F.1 · p. 27
  • Across three generation seeds, SkillPE Top Overall improves Creativity by +0.854 points and Overall by +0.200 points over Seed Skill.Both improvements are positive under every replicate, and Overall has a standard deviation of 0.02.Source: §F.1 · p. 27

F.2 Analysis of Benchmark-Native Score Changes

Skill evolution lowers some benchmark-native scores relative to Seed Skill, with decreases in VBench quality and semantic components and in StoryEval event completion. Some VBench dimensions improve, including dynamic degree for Top Overall.

F.2 Analysis of Benchmark-Native Score Changes

  • On MiniMax-H3/VBench, selected libraries reduce the native overall score by 1.30–2.12 points, with larger decreases in semantic than quality scores.Semantic decreases are −2.97 to −6.47, versus −0.79 to −1.32 for quality; Top Overall increases dynamic degree by 8.33 points.Source: §F.2 · p. 27
  • On StoryEval, selected libraries reduce native event-completion scores by 1.92–2.86 points relative to Seed Skill, with larger decreases on the hard subset.Hard-subset decreases are −3.39 to −4.83 points; the passage notes stronger cinematic treatment can occasionally interfere with faithful realization of requested events.Source: §F.2 · p. 28

F.3 Length Distribution of PE Result

The appendix reports prompt-length distributions and notes that final prompt lengths are comparable across methods after backbone-specific adaptation.

F.3 Length Distribution of PE Result

  • After shared backbone-specific adaptation, final prompt lengths are comparable across methods, with only minor differences.Table 8 presents the PE-result length distribution for each baseline.Source: §F.3 · p. 28

F.4 Robustness of Human Evaluation

The human evaluation was tested with two-way bootstrap resampling over prompts and annotators. All three final SkillPE libraries retained positive improvements over Seed-skill PE, with 95% confidence intervals above zero.

F.4 Robustness of Human Evaluation

  • 10,000 two-way bootstrap replicates independently resampled 40 prompts and ten annotators with replacement while preserving matched methods.The procedure accounts for uncertainty from both prompt and annotator sampling.Source: §F.4 · p. 28
  • All three final SkillPE libraries retained positive improvements over Seed-skill PE under the two-way bootstrap analysis.The 95% confidence intervals remained above zero.Source: §F.4 · p. 28
  • The reported uncertainty intervals were 95% confidence intervals from the bootstrap analysis.Table 9 reports pointwise 95% percentile confidence intervals.Source: §F.4 · p. 28

G Case Study

The case study shows that candidate skills trade off prompt fidelity, cinematic quality, narrative appeal, and creativity. Adaptive exploration offers conservative refinements and more divergent stylistic alternatives, while fine-grained evolution and skill selection address these competing outcomes.

G Case Study

  • Extreme achieved the highest StoryEval score (0.67) and creativity (7), combining dynamic camera movement with explicit mid-air fruit acquisition.Figure 4 presents examples from adaptive skill discovery for the monkey-and-fruit prompt.Source: G Case Study · p. 28, Figure 4 · p. 29
  • Bold reached cinematic quality (7) through a continuous push-in and atmospheric lighting, but omitted clear swinging and grabbing actions.Wilder instead emphasized close-ups, a shot transition, and a sound effect.Source: G Case Study · p. 28
  • Resonator and Dissonant augmentation raised narrative appeal from 3 to 5 and creativity from 2 to 5 through focused close-ups and tactile action details.These augmentations represent a more conservative refinement from the seed skill than the divergent alternatives.Source: G Case Study · p. 28
  • Consistent improvement across the official benchmark score and PF, CQ, NA, and CR scores was rare; more unconventional presentation could blur events or deviate from user intent.CQ, NA, and CR can also trade off, as the Wilder-level case improved narrative appeal and creativity while slightly hurting cinematic quality.Source: G Case Study · p. 28
  • The case study motivates fine-grained skill evolution and assessed skill selection to manage competing evaluation dimensions.R&D refines the seed conservatively, while divergents explore different degrees of presentation creativity.Source: G Case Study · p. 28

Related papers

Loading 2609.34335v1…