PaperlayerView exact revision on arXiv

Source-linked AI summary

SAGE: Self-Evolving Storyboard Skills via Attribution-Guided Rule Evolution

Maolin Ran, Xiaoyang Lu, Jiaqi Liu, Jian Wang, Weiwen Liu, Jianghao Lin, Yong Yu, Weinan Zhang

arXiv:2608.17468v1cs.AI

TL;DR

Professional storyboarding remains a bottleneck because directing expertise is tacit and difficult to acquire, refine, and inject into language-model workflows. SAGE evolves and routes directing rules from expert demonstrations, achieving 77.8 versus professional directors’ 77.1 on 18 episodes across three genres.

  • Problem

    Professional storyboarding remains a production bottleneck because its tacit directorial expertise is rarely articulated as explicit principles.

  • Method

    SAGE extracts content-free rules from expert screenplay-storyboard pairs, attributes localized feedback to adopted rules, and routes evolved rules through scenario packages.

  • Results

    77.8 versus professional directors’ 77.1 on 18 episodes across three genres, with attribution-guided iteration adding 3.6 points over rule warm-start.

  • Takeaways & Limitations

    Rule-level attribution reached a stable optimum whereas episode-level feedback declined, suggesting localized credit assignment is important for knowledge evolution beyond storyboarding.

  • Takeaways & Limitations

    One backbone was evaluated on only 16 of 18 episodes because of two routing failures.

Abstract

from arXiv · show

Storyboards turn screenplays into visual shot plans for automated short drama production. Professional storyboarding relies on tacit directorial expertise and remains an industrial bottleneck. Large language models can automate this step, but methods for supplying directing knowledge face three challenges: (1) Knowledge acquisition: the craft remains implicit in exemplars or must be written manually. (2) Knowledge refinement: authored knowledge is not evaluated against execution outcomes, and opaque generation prevents feedback attribution to the knowledge behind each decision. (3) Knowledge injection: injecting all knowledge exceeds usable context, while manual selection for every narrative group does not scale. We present SAGE (Skill with Attribution-Guided Evolution), a deployed framework that learns, attributes, evolves, and routes directing knowledge from expert demonstrations. SAGE derives rules that are independent of episode content by contrasting each training screenplay with its expert storyboard. During generation, the model records each narrative group's adopted rules. Combining these records with localized feedback enables targeted updates to individual rules. Evolved rules form scenario packages with a routing index, so each group retrieves only a bounded set appropriate to its situation without expert intervention. On 18 test episodes across three genres, SAGE scored 77.8 on a rubric validated by experts, versus 77.1 for professional directors. Deployed for 14 days on Virtual Film Studio, SAGE produced 1,344 narrative group outputs; 87.2 percent were accepted without substantive edits, and the production team recorded over 83 percent less authoring time per episode. We release PROSE, the first public dataset pairing screenplays with storyboards by professional directors across 68 episodes: https://github.com/creDreams/PROSE.

1 Introduction

Storyboards connect screenplays to video synthesis but remain a manual production bottleneck, while existing knowledge-injection methods struggle with acquisition, refinement, and scalable selection. SAGE addresses these challenges by extracting, attributing, evolving, and routing directing rules, achieving 77.8 versus professional directors’ 77.1 on 18 episodes across three genres.

  • Motivation: Manual storyboarding limits short-drama throughput even as automated film systems increasingly depend on storyboards to specify shot-level visual decisions for video synthesis.Storyboards determine shot content, scale, angle, and movement, constraining downstream fidelity, coherence, and cinematic expressiveness.
  • Method: SAGE makes generation traceable by recording adopted rules, enabling localized feedback to refine the individual knowledge responsible for each decision.Existing knowledge remains fixed or evaluates whole trajectories, whereas effective refinement requires attributing execution feedback to the responsible knowledge.
  • Method: SAGE consolidates evolved rules into scenario packages with a routing index, so each narrative group retrieves only situation-matched knowledge instead of manually selecting from thousands of entries.This addresses the context limits and scalability problems of injecting all knowledge or selecting it manually for every narrative group.
  • Method: SAGE extracts content-free directing rules by contrasting each training screenplay with its expert storyboard, addressing knowledge acquisition without manual rule authoring.Unlike few-shot exemplars, which encode craft implicitly, SAGE produces an inspectable, reusable rule representation.
  • Results: 77.8 versus 77.1: SAGE exceeded professional directors on 18 test episodes spanning three drama genres.Scores came from an LLM rubric whose agreement with three professional directors exceeded inter-human agreement, and SAGE also outperformed director-authored reasoning chains and adjacent-episode exemplars.

2 Related Work

Prior storyboard systems either coordinate agents, retrieve conventions, or keep cinematic knowledge implicit, while existing datasets omit directors’ pre-production decisions. SAGE instead treats directing knowledge as explicit, persistent, and rule-level optimizable context with attribution-guided updates.

  • LLM-based storyboard generation: Existing storyboard generators coordinate role-playing agents [40] [38], retrieve camera conventions [14], or encode cinematic knowledge implicitly through fine-tuning [53].These systems do not learn an explicit, inspectable knowledge base from professional storyboard demonstrations.
  • Storyboard datasets: PROSE releases directors’ original pre-production storyboards, unlike datasets whose annotations are reverse-engineered from finished videos [32] [53].This preserves decision traces needed for demonstration-supervised evolution.
  • Experiential knowledge in self-evolving agents: Self-Refine [20] and Reflexion [30] improve outputs or store trajectory reflections but do not accumulate knowledge that persists beyond the task.Unaided self-correction is also unreliable [13].
  • Natural-language component optimization: SAGE maintains rule-adoption records that assign credit to individual rules, enabling one round to revise, add, and retire different rules at once.This granularity keeps iteration productive where document-level optimization plateaus (§4.2).

3 Methodology

SAGE treats directing knowledge as an evolving, content-free rule base coupled to a fixed storyboard-generation workflow. It extracts and attributes rules from expert demonstrations, diagnoses them against localized feedback, and consolidates them for scenario-based retrieval.

  • Skill representation and learning objective: SAGE defines a skill as a fixed workflow paired with content-free executable rules, and optimizes those rules against expert storyboard alignment while evaluating quality separately.Rules specify narrative conditions and practices across five dimensions without copying concrete shot content or verbatim expert material; the alignment objective compares generated and expert storyboards, whereas the test metric is reference-free.
  • Overview: SAGE comprises evolution, consolidation, and inference stages that share narrative grouping, scenario routing, rule injection, attributed generation, and segment merging.Evolution refines per-episode rules, consolidation creates scenario packages with a routing index, and inference applies the packaged knowledge to unseen screenplays.
  • Attribution-guided evolution: Each evolution round extracts rules by contrasting screenplay groups with expert shots, records adopted rules during generation, evaluates localized deviations, and revises or adds rules through attribution-guided diagnosis.Misfiring rules are revised when adopted groups deviate, whereas coverage gaps induce new rules; only a capped number of rules may change per round.
  • Attribution-guided evolution: Rule-level attribution makes feedback actionable by linking each group’s adopted rules to dimension-level deviations, unlike episode-level feedback that cannot identify responsible rules.This mechanism converts undirected episode-level rewriting into targeted rule revision or rule induction and is identified as necessary for sustained improvement.
  • Consolidation: Consolidation compresses redundant and sometimes contradictory per-episode rule unions through similarity-based deduplication while preserving divergent practices as alternatives under shared conditions.Rules are merged within dimensions using a two-level cosine-similarity threshold of 0.90; conservative merging avoids irrecoverable loss of distinct knowledge.

4 Experiments

Across 18 held-out episodes from three professional short-drama series, SAGE achieved 77.8 overall quality, exceeding the professional-director reference at 77.1 and outperforming prompting and optimization baselines. Ablations and transfer experiments show that attribution stabilizes iterative rule evolution and that consolidated knowledge transfers across backbone LLMs.

  • Dataset and evaluation: The PROSE test set contained 18 held-out episodes spanning sci-fi suspense, urban romance, and emotional healing, scored across five storyboard-quality dimensions.The overall score averaged shot rhythm, visual description, shot scale, camera angle, and camera movement under a fixed reference-free rubric.
  • Main comparison: 77.8 overall made SAGE the only AI system to exceed the professional director’s 77.1 on the 18-episode test set.SAGE was also closest to the director on camera angle and camera movement, while visual description surpassed the director.
  • Ablation: Contrastive warm-starting delivered the largest single component gain, while attribution more than doubled the benefit of iteration over episode-level feedback alone.Attribution produced its largest gains on shot scale and visual description, where episode-level feedback cannot identify which rules require correction.
  • Evolution dynamics: With attribution, quality rose from 74.2 to 77.8 by round 5 and remained stable through round 10, whereas without attribution it peaked at 75.4 and fell to 73.8.Both settings shared the same round-1 rule set, isolating attribution’s effects on improvement magnitude and stability.
  • Cross-model transfer: Injecting unchanged Claude-evolved scenario packages improved every transfer target by 8.1% to 21.2% relative, with GPT-5.4 approaching the source model’s quality.Weaker backbones benefited more, visual description transferred most universally, and GLM-5.2 used 16 episodes because of two routing failures.

5 Production Deployment

SAGE was deployed in VFS to test whether offline gains translate into production value. Across three ongoing dramas, it achieved high acceptance and substantially reduced authoring time while keeping rejected work locally traceable.

  • Deployment setting: SAGE was deployed in VFS using the same framework trained on a larger proprietary corpus of director demonstrations.The deployment tested whether offline gains translated into production value on an in-house creation platform used by over a thousand creators.
  • Evaluation protocol: Acceptance was defined at the narrative-group generation unit, with only outputs requiring no substantive downstream production edits counted as accepted.Changes limited to asset references, formatting, punctuation, or wording were treated as nonsubstantive.
  • Production results: 87.2% of 1,344 narrative-group outputs were accepted without substantive edits, with rates ranging from 82.3% to 100.0% across three productions.Table 4 reports 1,172 accepted outputs; acceptance was measured from platform interaction logs.
  • Production results: Authoring time fell from over one hour to roughly 10 minutes per episode, an approximately sixfold acceleration tracked by the production team.The turnaround was measured over the same deployment period as the acceptance logs.
  • Deployment mechanism: Rejected outputs require only local narrative-group regeneration and remain traceable to their adopted rules through inference-time records.These properties bound residual manual effort without requiring episode-level revision passes.

6 Conclusion

SAGE evolves directing knowledge from expert demonstrations, routes feedback to individual rules, and deploys the resulting rules as scenario packages without a reference. Its iteration study shows that rule-level attribution supports stable knowledge evolution, whereas episode-level feedback declines, suggesting localized credit assignment is architecturally necessary.

  • 6 Conclusion: SAGE learns directing knowledge from PROSE demonstrations, routes feedback through rule-adoption records, and consolidates evolved rules into scenario packages for reference-free deployment.This removes dependence on exhaustive manual externalization of experts’ tacit directorial knowledge.
  • 6 Conclusion: Rule-level attribution reached a stable optimum, whereas episode-level feedback declined, showing that knowledge evolution depends on localizing credit assignment to individual items.The conclusion characterizes this requirement as architectural rather than specific to storyboarding or another single domain.

A Rule Consolidation Details

SAGE consolidates learned rules into coherent, well-separated scenario clusters that become routable packages organized by professional dimension. The packages mix rules across series while preserving situation-specific dimensional signatures rather than balancing dimensions artificially.

  • Rule Consolidation Details: Clusters mix rules from all three series, indicating that their conditions describe narrative situations rather than source-drama idiosyncrasies.One example package contains 38 rules contributed by all three series.
  • Rule Consolidation Details: Consolidation preserves each situation’s dimensional signature, with camera angle dominating power-asymmetry packages while shot scale and rhythm dominate emotional-release packages.The distribution across professional dimensions is intentionally uneven rather than artificially balanced.
  • Rule Consolidation Details: Rule-condition embeddings form coherent, well-separated clusters, each converted into a routable scenario package organized by professional dimension.Figure 6 illustrates a package containing 38 rules, with representative rules grouped by dimension.

B Scenario-Aware Inference Example

A held-out test episode demonstrates scenario-aware inference: a two-person confrontation retrieves three complementary scenario packages, injects their rules into generation, and records adoption without expert reference. The records expose which package shaped each shot, enabling rejected outputs to be traced to the responsible rule.

  • B Scenario-Aware Inference Example: Figure 7 traces a held-out test episode in which a two-person confrontation retrieves its top-3 scenario packages, injects their rules, and records rule adoption without expert reference.Each shot is marked with the primary package whose rules it adopted, with one representative rule shown per package.
  • B Scenario-Aware Inference Example: Adoption records make the division visible after generation, allowing a rejected output to be traced to the rule that shaped it.The records link generated shots back to the packages whose rules informed them.
  • B Scenario-Aware Inference Example: The three retrieved packages are complementary: they govern power-asymmetry angles, verbal-exchange rhythm, and physical-intrusion framing across different shots.Their rules act on different shots of the same segment rather than redundantly controlling the entire segment.

C Alignment Review Skill

The alignment review skill compares generated storyboards with hidden director references across five professional dimensions, treating the reference as the sole correct target. It produces a diagnosis-oriented signal by ignoring surface variation and requiring cited deviations at divergent shots.

  • Review design: The review skill compares generated storyboards with director references across five professional dimensions, while the generator never sees the reference.The director storyboard is treated as the gold-standard target for every dimension.
  • Diagnostic signal: The comparison discards layout, column order, and wording differences so detected deviations reflect directorial decisions rather than formatting artifacts.This isolates substantive storyboard discrepancies for rule-level diagnosis.
  • Diagnostic signal: The report records only deviations, and each must cite the shot numbers where the generated and director storyboards diverge.Praise and hedging are suppressed, while uncited deviations cannot support attribution.

D Quality Scoring Rubric

The quality metric f_qual evaluates storyboards against absolute professional standards, emphasizing emotion and story for vertical short drama. It averages five consistently structured, evidence-anchored dimensions scored on 100-point scales with shot-specific justifications.

  • Rubric basis: f_qual scores storyboards on absolute professional standards without reference, prioritizing emotion and story over rhythm and continuity in high-density vertical short drama.The rubric reflects Walter Murch’s editing priorities, while accounting for the importance of opening seconds for retention.
  • Scoring structure: Each of five dimensions uses a 100-point scale divided into five 20-point bands, and the overall score is their unweighted mean.Every band is anchored to observable evidence, and the shared structure keeps scores comparable across dimensions.
  • Scoring prompt: Figure 9 presents the translated and condensed scoring prompt, illustrating the five-band rhythm rubric shared across all five dimensions.The other dimensions are abridged to their core criteria in the figure.
Loading 2608.17468v1…