Source-linked AI summary
DramaBench: A Six-Dimensional Evaluation Framework for Drama Script Continuation
Shijian Ma, Yunqi Huang, Yan Lin
TL;DR
Existing benchmarks do not comprehensively assess the multiple qualities required for drama script continuation. DramaBench evaluates continuation across six dimensions using rule-based analysis and structured LLM labeling, and reports model specialization alongside statistical and human validation. Its stated limitations include evaluating continuation rather than long-span full-script coherence and validating only 17% of the dataset.
Problem
Existing benchmarks lack quantitative, multidimensional evaluation specialized for drama script continuation, while human evaluation is costly and slow.
Source: §1 · p. 1, §4.1 · p. 4Method
DramaBench combines rule-based format analysis with structured LLM labeling and statistical metrics across six independent drama-continuation dimensions.
Source: §1 · p. 1Results
No model excels universally: GPT-5.2 leads in robustness, Qwen3-Max in emotional depth, and Gemini-3-Pro in conflict handling; 65.9% of 252 comparisons were significant.
Source: §10 · p. 10Takeaways & Limitations
DramaBench provides dimension-specific evaluation and actionable feedback for targeted model improvement.
Source: §10 · p. 10Human validation covered only 17% of the dataset, and the benchmark evaluates continuation rather than long-span full-script coherence.
Source: Limitations · p. 10
Abstract
from arXiv · showhide
Drama script continuation requires models to maintain character consistency, advance plot coherently, and preserve dramatic structurecapabilities that existing benchmarks fail to evaluate comprehensively. We present DramaBench, the first large-scale benchmark for evaluating drama script continuation across six independent dimensions: Format Standards, Narrative Efficiency, Character Consistency, Emotional Depth, Logic Consistency, and Conflict Handling. Our framework combines rulebased analysis with LLM-based labeling and statistical metrics, ensuring objective and reproducible evaluation. We conduct comprehensive evaluation of 8 state-of-the-art language models on 1,103 scripts (8,824 evaluations total), with rigorous statistical significance testing (252 pairwise comparisons, 65.9% significant) and human validation (188 scripts, substantial agreement on 3/5 dimensions). Our ablation studies confirm all six dimensions capture independent quality aspects (mean | r | = 0.020). DramaBench provides actionable, dimensionspecific feedback for model improvement and establishes a rigorous standard for creative writing evaluation.
1 Introduction
Drama script continuation demands multiple forms of consistency and structure, yet existing benchmarks lack quantitative, drama-specific multidimensional evaluation. DramaBench evaluates continuation across six dimensions using structured LLM labels and statistical metrics, with validation and analyses intended to support reproducible, actionable assessment.
2.1 Story and Drama Generation
- Existing approaches evaluate full-script generation or general story and NLG qualities, but do not provide quantitative assessment of screenplay-specific continuation requirements.The cited benchmarks include story coherence and general dimensions, rather than screenplay-specific format, character voice, or dramatic structure.Source: §1 · p. 1, §2.1 · p. 2, §2.2 · p. 3
1 Introduction
- DramaBench evaluates script continuation across six dimensions, using structured LLM labels converted into statistical metrics rather than direct quality scores.The framework combines rule-based format analysis with LLM labeling for other dimensions.Source: §1 · pp. 1–2
- Validation includes 252 significance tests, dimension-independence ablations, and human evaluation on 188 scripts, with substantial agreement on three of five dimensions.The reported mean absolute correlation between dimensions is 0.014.Source: §1 · p. 2
- The paper reports an error taxonomy covering 10,850 errors and 24 case studies, alongside model-specific profiles intended to identify improvement opportunities.The authors also state that evaluation scripts, metrics, and analysis code will be released for reproducible research.Source: §1 · p. 2
- No model excels across every dimension: GPT-5.2 leads in robustness, Qwen3-Max in emotional depth, and Gemini-3-Pro in conflict management.All models achieve near-perfect format compliance, while logic consistency varies substantially.Source: §1 · p. 2
2.2 Multi-Dimensional NLG Evaluation
- Surveys report reliability concerns with direct LLM scoring, while DramaBench instead uses structured labeling; labels can also serve as training data for model improvement.The framework aims for reproducibility and interpretability by tracing metrics to labeled instances.Source: §1 · p. 1, §2.2 · p. 3
2.3 Evaluation Benchmarks
- DramaBench positions itself as combining large-scale quantitative evaluation, drama-specific metrics, significance testing, and human–LLM agreement analysis for screenplay continuation.Its positioning describes 8,824 samples.Source: §2.3 · p. 3
3 Dataset
DramaBench builds continuation tasks from curated English Fountain-format drama scripts, splitting them at scene boundaries when possible. Its evaluation combines rule-based format checks with structured LLM labeling and statistical metrics across narrative, character, emotion, logic, and conflict dimensions.
3.1 Data Collection and Processing
- The benchmark uses 1,103 professionally written English short-form drama scripts in Fountain format, filtered to remove duplicates, truncations, and formatting corruption.Scripts must also contain dialogue and character interactions and fall between 27 and 269 lines.Source: §3.1 · p. 3, §3.4 · pp. 3–4
3.2 Scene-Boundary-Aware Splitting
- Splits target the 50% line-count midpoint, then search a ±20% window for scene headings; a scene boundary is used in 69.5% of cases.When no boundary is found, the midpoint is used, yielding continuations that begin at natural narrative breakpoints.Source: §3.2 · p. 3
3.3 Dataset Statistics
- Contexts average 381 tokens and continuations 401 tokens, requiring moderate-context consistency while generating substantial creative content.Table 1 summarizes dataset statistics and identifies deterministic scene-boundary splitting.Source: §3.3 · p. 3, Table 1 · p. 3
4.1 Overview: LLM Labeling + Statistical Analysis
- LLMs act as structured data annotators, assigning categorical labels that statistical formulas convert into quality metrics.The approach is designed for reproducibility, interpretability, and reuse of labels as training data.Source: §4.1 · p. 4
4.2.1 Format Standards (Rule-Based)
- Format Standards uses Fountain rules to measure format errors, novelization, and dialogue-action balance without an LLM.The stated targets are under 1% format errors, a Novelization Index under 0.35, and a Dialogue-Action Ratio of 1.0–2.0 for short drama.Source: §4.2.1 · p. 4
4.2.2 Narrative Efficiency
- Narrative Efficiency classifies extracted story beats as Driver, Static, or Redundant and measures effective narrative rate and drivers per page.The dimensions target plot progression density and padding behavior.Source: §4.2.2 · p. 4
4.2.3 Character Consistency
- Character Consistency assesses scene-level emotional shifts and complex emotions, then classifies dialogue as in-character, neutral, or out-of-character against persona profiles.Its metrics include an Arc Score, Complexity Ratio, OOC Rate, and Voice Distinctiveness.Source: §4.2.3 · pp. 4–5
4.2.5 Logic Consistency; 4.2.6 Conflict Handling
- Logic Consistency extracts context constraints and classifies each as violated, maintained, or irrelevant, while Conflict Handling scores escalation, twists, pauses, resolution, and dropped conflicts.The resulting metrics include Logic Break Rate and Context Coherence for logic, plus Conflict Score and Drop Rate for conflict handling.Source: §4.2.5 · p. 5, §4.2.6 · p. 5
5 Experimental Setup
The evaluation tests eight state-of-the-art language models on shared script contexts and continuation instructions. LLM-labeled dimensions use Qwen3-Max as evaluator, while format compliance is checked deterministically.
5.1 Models Evaluated
- The study evaluates 8 state-of-the-art language models representing diverse architectures and training paradigms, including DeepSeek v3.2, GLM-4.6, Gemini 3 Pro Preview, and Kimi K2 Thinking.Source: §5.1 · p. 5
5.2 Evaluation Process
- All models receive the same context and instruction to continue each script while maintaining established characters, plot, and style.Qwen3-Max evaluates LLM-labeled dimensions; Format Standards uses deterministic rule-based analysis.Source: §5.1 · p. 5, §5.2 · p. 6
- Format analysis has no cost, while LLM-labeled dimensions incur API costs based on evaluator token usage.The authors state that evaluation prompts, extracted labels, and scripts are deterministic and will be released.Source: §5.2 · p. 6
6 Results
Across six dimensions, no model dominates: GPT-5.2 leads in robustness, Qwen3-Max in emotional depth, and Gemini-3-Pro in conflict handling. Results range from universal format compliance to greater variation in logic consistency.
6.3.1 Format Standards
- All models had a 0.0% mean format error rate, with 228 warnings across 8,824 evaluations (2.6% sample error rate).GPT-5.2 had a Novelization Index of 0.11, while DeepSeek v3.2 scored 0.24.Source: §6.3.1 · p. 6
6.3.2 Narrative Efficiency
- Mean ENR was 93.3%, but its distribution was left-skewed (−1.81); beat density averaged 3.09 per page, from 3.52 for GPT-5.2 to 2.43 for GLM-4.6.Beats per Page was the most differentiating metric in statistical testing.Source: §6.3.2 · p. 6
6.3.3 Character Consistency
- Mean OOC rate was 0.97%, and 87.1% of scripts had perfect character consistency; the distribution was highly right-skewed (+11.06).Models averaged 84.1% in-character dialogue, with a 12.9% outlier rate for OOC errors.Source: §6.3.3 · p. 6
6.3.4 Emotional Depth
- 90.3% of continuations included emotional shifts, and 97.5% of scenes included complex emotions, averaging 2.57 complex emotional moments per continuation.Arousal shifts occurred in 62.8% of cases versus 29.0% for valence shifts.Source: §6.3.4 · pp. 6–7
6.1 Overall Performance
- No model dominates all six dimensions: GPT-5.2 ranks first in Narrative, Character, and Logic, while Qwen3-Max leads Emotional Depth and Gemini-3-Pro Conflict Handling.All models achieve 0% format error rate, but rankings vary by dimension.Source: §6.1 · p. 6, Table 2 · p. 7
6.2 Statistical Significance Testing
- 65.9% of 252 comparisons were significant; Beats per Page differed most (26/28 significant, 20 large effects), while Conflict Score differed least (8/28).The tests used Mann-Whitney U comparisons with Benjamini-Hochberg FDR correction at q = 0.05.Source: §6.2 · p. 6, Table 3 · p. 7
6.3.5 Logic Consistency
- Mean logic break rate was 3.6%; GPT-5.2 and DeepSeek v3.2 had rates of 2.0–2.3%, while GLM-4.6 had the highest rate at 5.3%.Models maintained an average of 6.88 facts per script; the logic dimension had the highest variance across models.Source: §6.3.5 · p. 7
6.3.6 Conflict Handling
- Models chose Escalation in 79.1% of scripts, with a 1.5% drop rate; 75.6% of continuations introduced or developed secondary conflicts.The authors relate escalation preference to serial drama conventions of intensifying conflict mid-episode.Source: §6.3.6 · p. 7
7 Analysis
Case studies, error profiles, and cross-dimensional comparisons show distinct strengths and weaknesses across models. The analysis also finds that script-quality dimensions are largely independent, with format compliance separate from content quality.
7.1 Case Studies
- The case studies contrast an ENR = 1.0 GPT-5.2 continuation with 18 driver beats against a DeepSeek v3.2 continuation with ENR = 0.33 and 67% static beats.The GPT-5.2 continuation advanced through discovery, betrayal, planning, and confrontation in 62 lines; the DeepSeek continuation dwelled on expressions and emotional states.Source: §7.1 · pp. 7–8
- Other cases show a MiniMax M2 continuation contradicting the established surgery location and a Claude Opus 4.5 continuation maintaining all 31 dialogue lines in character.The Logic Consistency case study contrasts preserved context constraints with spatial or temporal violations.Source: §7.1 · pp. 7–8
7.2 Error Taxonomy
- Error profiles differ by model: GPT-5.2 has minimal errors overall, Qwen3-Max has 243 dialogue-imbalance occurrences, and GLM-4.6 has 102 excessive-prose occurrences and the most logic violations.The analysis presents these profiles as actionable improvement targets.Source: §7.2 · p. 8
7.3 Cross-Dimensional Insights
- Format Standards correlate near-zero with content dimensions (mean |r| = 0.040), while Narrative Efficiency, Character Consistency, and Logic Consistency show weak positive correlations (r ≈ 0.05).The latter association is described as a robustness cluster.Source: §7.3 · p. 8
7.2 Error Taxonomy
- The error taxonomy covers 10,850 errors; Dialogue-Action Imbalance and Low Information Gain are the most common among the top ten types.The figure abbreviates Out-of-Character as OOC and Effective Narrative Rate as ENR.Source: §7.2 · p. 8, Figure 3 · p. 9
7.3 Cross-Dimensional Insights
- Emotional Depth and Conflict Handling are independent of other dimensions, with correlations below 0.02, indicating distinct capabilities that can be optimized separately.The figure compares the six dimensions’ model-performance profiles and reports no universal leader.Source: Figure 2 · p. 8, §7.3 · p. 9
8 Ablation Studies
The analyses support treating the five content dimensions as distinct, non-redundant measures, while human agreement varies across dimensions. Format Standards was excluded from the correlation analysis because every model achieved full compliance.
8.1 Dimension Independence Validation
- The five content dimensions show near-zero pairwise correlations (mean |r| = 0.014), supporting their distinctness; Format Standards was excluded because all models had 100% compliance.The authors conclude that removing any content dimension would lose information.Source: Figure 4 · p. 9, §8.1 · p. 9
8.2 Human-LLM Agreement Analysis
- Human annotations on 188 scripts agreed moderately to substantially with the evaluator on Logic, Emotional Depth, and Conflict, but showed no significant correlation for Narrative Efficiency or Character Consistency.The authors advise caution when interpreting model rankings on the latter two dimensions.Source: Table 5 · p. 9, §8.2 · p. 9
9 Discussion
The discussion argues that dimension-specific assessment and error analysis reveal actionable model-improvement targets, while noting evaluator, language, and genre limits and proposing future extensions.
9.1 Implications
- Dimension-specific evaluation can reveal contrasts—such as strength in conflict management alongside weakness in character consistency—that aggregate scores mask.The discussion presents multi-dimensional evaluation as essential for retaining these distinctions.Source: §9.1 · p. 9
- Error taxonomies and case studies identify actionable, dimension-specific targets for model improvement.The discussion gives GLM-4.6’s logic violations and Qwen3-Max’s dialogue imbalance as examples.Source: §9.1 · p. 9
- Extracted labels such as out-of-character dialogue, redundant beats, and logic violations can serve as negative samples for Direct Preference Optimization.This proposed data loop is intended to enable continuous model improvement.Source: §9.1 · p. 10
9.2 Limitations
- Human–LLM agreement analysis identifies systematic Qwen3-Max evaluator bias on narrative efficiency and character consistency.The discussion proposes multi-evaluator ensemble voting to mitigate single-model bias.Source: §9.2 · p. 10
- DramaBench covers English scripts only; multilingual evaluation requires language-specific format parsers and evaluators.Source: §9.2 · p. 10
- The dataset focuses on short-form drama; full-length features, comedy, and experimental formats may require modified dimensions.Source: §9.2 · p. 10
9.3 Future Work
- Future work proposes multi-LLM consensus voting, annotation examples in prompts, larger human-validation studies, and extensions to other creative-writing domains.The suggested larger study would cover 500+ scripts and 5+ annotators.Source: §9.3 · p. 10
10 Conclusion
DramaBench evaluates drama script continuation across six dimensions using LLM labeling and statistical analysis. Across eight models, strengths vary by dimension, and statistical and human validation support the framework.
10 Conclusion
- DramaBench is presented as the first large-scale benchmark for drama script continuation across six independent dimensions.Its LLM Labeling + Statistical Analysis framework is intended to provide objective, reproducible evaluation and actionable feedback.Source: §10 · p. 10
- Across 8 models and 1,103 scripts, GPT-5.2 leads in robustness, Qwen3-Max in emotional depth, and Gemini-3-Pro in conflict handling.Source: §10 · p. 10
- 65.9% of 252 comparisons were significant; ablations reported mean correlation |r| = 0.020, and human validation found substantial agreement on 3/5 dimensions.Source: §10 · p. 10
Ethics Statement
The ethics statement describes the dataset and evaluation practices, while the limitations note boundaries in continuation scope, format, cost, and human-validation coverage.
Ethics Statement
- Scripts came from publicly available datasets; the authors report excluding personal information, unauthorized copyrighted material, and sensitive content, and using official APIs under usage agreements.They also acknowledge potential LLM-evaluator bias and recommend ensembles for production use.Source: Ethics Statement · p. 10
Limitations
- DramaBench evaluates continuation rather than long-span full-script coherence, assumes Fountain formatting, scales evaluation cost with script and model counts, and human validation covered 17% of the dataset.The authors state that larger human-validation sets would strengthen claims.Source: Limitations · p. 10
A Extended Case Studies
The appendix presents qualitative case studies illustrating success and failure patterns across narrative efficiency, character consistency, and emotional depth.
A.1 Narrative Efficiency
- 8 of 12 beats are static in one failure case, contrasting with a success case where every beat advances the plot.Source: §A.1 · p. 12
A.2 Character Consistency
- A character-consistency success case maintains a grief-driven, angry, vengeful persona throughout the continuation.The case establishes Zhou Sheng as grief-stricken, vengeful, and determined.Source: §A.2 · p. 12
A.3 Emotional Depth
- 7 instances of opposing emotions illustrate emotional complexity, while a contrasting failure case stays flat from panic through terror.Source: §A.3 · p. 12
A.4 Logic Consistency
The appendix contrasts escalating conflicts with abandoned ones and reports uniform Fountain-format compliance across all eight models.
A.4 Logic Consistency
- Detailed success and failure patterns for Logic Consistency are presented in main-text Table 4.Source: §A.4 · p. 12
A.5 Conflict Handling
- Successful continuations escalated an initial assassination threat with three secondary conflicts, raising stakes and adding internal conflict.The examples include a marriage deadline and opposition to the marriage.Source: §A.5 · p. 13
- A failed continuation dropped the conflict through a 20-year time skip, leaving it unresolved while skipping the daughter’s fate and the father’s guilt.The initial conflict involved the father selling his daughter and the mother’s suicide.Source: §A.5 · p. 13
A.6 Format Standards
- All 8 models had a 0.0% format error rate on Fountain screenplay format, which the authors describe as solved through prompt engineering.The authors distinguish structural compliance from semantic quality across the other five dimensions.Source: §A.6 · p. 13