PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Unified Evaluation of Prompt Enhancers for Video Generation

Yawen Shao, Yubo Zhu, Ziyun Dai, Zixun Fang, Kai Zhu, Zeyinzi Jiang, Yufeng Ai, Siyang Sun, Haolan Xue, Yu Shang, Yuxiang Bao, Zoubin Bi, Jingming Luo, Jie Xiao, Chaojie Mao, Zhehan Kan, Hongchen Luo, Yu Liu, Sheng Zhong, Wei Tong, Xueyang Fu, Yang Cao, Wei Zhai, Zheng-Jun Zha

arXiv:2610.11736v1cs.CV

TL;DR

Existing PE evaluation depends on rendered videos, adding cost and latency while making enhancer quality difficult to separate from generator behavior. PEBench directly evaluates prompts across three enhancement settings with modality-aware evidence and rubrics; its results show an emerging shift toward cinematic planning and alignment with expert judgments.

  • Problem

    Rendered-video evaluation of prompt enhancers adds computational and human costs and confounds PE quality with downstream generator behavior.

    Source: §1 · p. 1

  • Method

    PEBench directly evaluates T2V-PE, I2V-PE, and R2V-PE prompts using modality-aware evidence and criterion-specific rubrics.

    Source: §1 · p. 2, §3.1 · p. 3, §3.4 · p. 6

  • Results

    Across tasks, enhancement is shifting toward intent-preserving cinematic planning, caption-reconstruction favors cinematic coverage, and forward-refinement favors semantic fidelity or internal coherence; Spearman correlations exceed 0.8 with expert rankings.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Prompt-level PEBench scores reliably reflect downstream utility across generators, aligning with expert rankings of enhanced prompts and videos from Wan3.0 and MiniMax-H3.

    Source: §1 · p. 2

    I2V-PE and R2V-PE comparisons include fewer baselines because few methods support multimodal enhancement and few proprietary systems offer accessible interfaces.

    Source: F Limitations · p. 27

Abstract

from arXiv · show

Modern video generators can realize increasingly complex visual narratives, positioning the prompt enhancer (PE) as a critical bridge from concise user instructions and multimodal references to structured cinematic plans. However, existing PE evaluation relies on rendered videos, imposing substantial computational and human costs, slowing PE training and iteration, and conflating PE quality with downstream generator behavior. To address this gap, we introduce PEBench, the first unified benchmark for direct PE evaluation across text-to-video, image-to-video, and reference-to-video prompt enhancement. It comprises 1,100 expert-verified cases and 1,005 visual assets, spanning 35 fine-grained tasks with diverse temporal, cinematic, audiovisual, and multi-reference requirements. In addition, we develop PEBench evaluation, an evidence-grounded framework that combines modality-aware fact extraction with rubric-based assessment across 24 criteria. Our systematic evaluation of representative open- and closed-source PE methods reveals an emerging shift from fine-grained descriptive expansion toward intent-preserving cinematic planning, while the caption-reconstruction and forward-refinement methods show complementary strengths in cinematic coverage and semantic fidelity or internal coherence, respectively. Human validation shows that PEBench scores align closely with expert judgments of enhanced prompts and downstream videos from Wan3.0 and MiniMax-H3, indicating that prompt-level evaluation reliably reflects downstream utility.

1 Introduction

Concise or ambiguous instructions can leave video generators uncertain about user intent and reference roles, while video-based PE evaluation is costly and entangles enhancer quality with generator behavior. PEBench evaluates prompts directly across three settings and reports distinct capability profiles alongside alignment with expert judgments.

1 Introduction

  • Ambiguous instructions can obscure subject-action bindings, event order, and reference roles, while video-based evaluation costs time and compute and confounds enhancer and generator errors.Direct prompt evaluation is motivated as a way to provide PE-specific feedback independently of generator behavior.Source: §1 · pp. 1–2
  • PEBench unifies T2V-PE, I2V-PE, and R2V-PE in 1,100 expert-verified cases with 1,005 assets across 35 fine-grained task types.The benchmark covers text-only, first-frame, and multi-reference conditioning.Source: §1 · p. 2, Figure 1 · p. 2
  • Its evidence-grounded evaluation combines modality-aware fact extraction with rubric-based assessment across 24 criteria and three dimensions.The dimensions are Semantic Fidelity, Internal Consistency, and Directorial Creation.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • Evaluated systems show a shift toward intent-preserving cinematic planning, complementary strengths across enhancement strategies, and scores aligned with expert judgments of prompts and generated videos.The reported alignment covers downstream videos from Wan3.0 and MiniMax-H3.Source: §1 · pp. 2–3

2 Related work

Video prompt enhancers use training-based and inference-time strategies, while existing video-generation benchmarks assess rendered outputs rather than enhanced prompts. PEBench instead provides direct, standardized prompt-level evaluation across conditioning settings.

2 Related work

  • Training-based enhancers learn rewriting from supervision or preferences, while inference-time methods retrieve examples, verify and correct prompts, or use multimodal workflows.Examples include VPO [6], Prompt-A-Video [9], RAPO [8], SCMaPR [7], LTX-2.5-PE, and H3-Context-IR.Source: §2 · p. 3
  • Existing video benchmarks expanded to reference-conditioned, multi-shot, audio-video, and streaming generation, but evaluate rendered videos rather than enhanced prompts.Using them for PE assessment requires generation and entangles enhancer performance with downstream generator behavior.Source: §2 · p. 3
  • PEBench assesses enhanced prompts directly across T2V-PE, I2V-PE, and R2V-PE using standardized rubrics covering 24 criteria.This enables fine-grained assessment independent of downstream video generation.Source: §2 · p. 3

3 PEBench

PEBench defines direct prompt evaluation for text-to-video and visually conditioned enhancement, constructs a diverse benchmark through staged synthesis and curation, and applies evidence-grounded rubrics across three evaluation dimensions.

3.1 Task Definition

  • PEBench evaluates enhanced prompts directly for T2V, I2V, and R2V, assessing preservation of intent and conditioning constraints, internal consistency, and detail for the intended video.The settings cover text-only prompts, first-frame image conditioning, and multiple reference assets with specified roles.Source: §3.1 · pp. 3–4

3.2 Data Construction

  • A four-stage pipeline designs a task taxonomy, synthesizes and diversifies prompts, acquires reference assets, and verifies and curates cases.The process combines expert design and curation with model-based screening; ten domain experts refine verified cases for clarity and difficulty.Source: §3.2 · p. 4

3.3 Data Statistics

  • The 1,100 cases span 35 task types, 12 languages, 8 audio domains, prompts of 6–4,120 tokens, 1–13 shots, and 1,005 visual assets.The cases also vary in genres, subjects, scenes, visual styles, cinematic requirements, and image, video, and hybrid reference configurations.Source: §3.3 · p. 5, Figure 3 · p. 5

3.4 Evaluation Framework

  • Evaluation uses modality-aware evidence extraction followed by a unified evaluator applying fixed criterion-specific rubrics across 24 criteria.The criteria assess Semantic Fidelity, Internal Consistency, and Directorial Creation; shared evidence and rubrics support interpretable comparisons within each task setting.Source: §3.4 · pp. 5–6

4 Experiments

Experiments compare ten prompt enhancers and the original-prompt baseline across text-to-video, image-to-video, and reference-to-video tasks. Results show task-specific capability profiles, persistent complexity-related challenges, and close alignment between PEBench scores and expert rankings.

4.2 Main Results

  • Prompt enhancement is shifting from descriptive expansion toward intent-preserving cinematic planning as video generators support richer structured instructions.Internal Consistency varies by less than 9 points within each task, while Semantic Fidelity and Directorial Creation each span more than 70 points on T2V-PE.Source: §4.2 · pp. 6–7
  • No PE method dominates every task and dimension: H3-Context-IR leads all three dimensions on I2V-PE and R2V-PE, while T2V-PE leaders differ.Table 2 reports performance across all three tasks.Source: §4.2 · p. 6, Table 2 · p. 7
  • WanPE-397B and WanPE-35B lead Directorial Creation with 78.97 and 78.15; SCMaPR leads Semantic Fidelity with 87.92, while VPO leads Internal Consistency with 95.07.SCMaPR and VPO have lower Directorial Creation scores of 49.18 and 33.07, respectively; the authors link these profiles to complementary training strategies.Source: §4.2 · pp. 7–8

4.3 Analysis

  • Cases with three or more event stages trail one-stage cases by 3.67, 3.92, and 1.98 points in SF, IC, and DC; requested-shot gaps are larger.For requested shots, the corresponding gaps are 9.97, 7.61, and 5.52 points, indicating greater demands from multi-shot coordination.Source: Table 3 · p. 8, §4.3 · p. 8
  • On R2V-PE, cases with three or more images trail single-image cases by 7.42, 4.39, and 2.71 points in SF, IC, and DC.Hybrid inputs score 45.62 in SF and 55.69 in DC, below both single-modality settings, but reach the highest IC at 97.09.Source: Table 4 · p. 8, §4.3 · p. 8
  • Original prompts score 100.00 in Semantic Fidelity but 33.56 in Directorial Creation on T2V-PE, and longer outputs do not consistently improve Directorial Creation.Across tasks, the longest output quartile trails the shortest by 5.19–10.09 points; the authors recommend structured, task-relevant specifications rather than mere lengthening.Source: §4.3 · p. 8
  • Qualitative failures include visual-style deviation, subject–action binding errors, language or speaker attribution errors, narrative truncation, and incorrect multi-reference grounding.The corresponding Wan3.0 videos visibly reflect these errors.Source: §4.3 · p. 8, Figure 4 · p. 9

4.4 Human Alignment

  • PEBench aligns more closely with expert enhanced-prompt rankings than direct VLM scoring, with dimension-averaged gains of 0.333–0.358 across tasks.Correlations with expert rankings of Wan3.0 and MiniMax-H3 videos range from 0.809 to 1.000.Source: Table 5 · p. 9, §4.4 · p. 9

5 Conclusion

PEBench unifies direct prompt-enhancer evaluation and reports complementary method strengths, persistent complexity challenges, and alignment with expert judgments of prompts and downstream videos.

5 Conclusion

  • PEBench evaluates T2V-PE, I2V-PE, and R2V-PE using 1,100 expert-verified cases, 1,005 visual assets, 35 tasks, and 24 evidence-grounded rubrics.The framework combines modality-aware evidence with criterion-specific assessment and avoids costly video rendering.Source: §5 · p. 9
  • Evaluation reveals a shift from descriptive expansion toward intent-preserving cinematic planning, with caption-reconstruction and forward-refinement methods showing complementary strengths.The conclusion suggests combining cinematic knowledge with semantic and logical safeguards.Source: §5 · p. 9
  • Performance degradation under temporal and multi-reference complexity highlights persistent challenges, while scores align with expert judgments of enhanced prompts and downstream videos.The conclusion presents this alignment as confirming the practical relevance of direct prompt-enhancer evaluation.Source: §5 · p. 9

Appendix Overview

  • The appendices cover data, evaluation, experiments, expert annotation, ethics, privacy, licensing, and limitations concerning benchmark scale and future expansion.The overview identifies data screening and release policy among the ethics, privacy, and licensing topics.Source: Appendix Overview · p. 13

A.1 Data Design Details

PEBench’s data design spans three prompt-enhancement settings, a hierarchical task taxonomy, and varied scenario, language, temporal, and cinematic dimensions.

A.1 Data Design Details

  • The benchmark includes 500 text-only T2V-PE cases, 300 I2V-PE cases paired with a first-frame image, and 300 R2V-PE cases conditioned on one or more visual assets.R2V references can serve roles such as subject identity, scene, style, motion, cinematography, or audiovisual rhythm.Source: §A.1 · p. 13
  • Its taxonomy organizes 35 fine-grained task types under nine high-level capability categories, with each task assigned to one parent category.The categories span capabilities including visual, narrative, cinematic, speech, music, and sound.Source: §A.1 · p. 13, Table 6 · p. 13
  • Scenario composition varies five subject categories, eight scene domains, seven visual-style classes, and eight explicit audio domains.The listed categories include humans, animals, objects, and fantasy beings; scene and style categories range from domestic to otherworldly and photorealistic to anime.Source: §A.1 · p. 13
  • Prompts span 12 languages, concise concepts to detailed scripts, and single shots to as many as 13 structured shots, with varied camera, lens, composition, focus, transition, lighting, and color controls.Cinematic dimensions include shot scale, viewpoint, lens family, and camera motion.Source: §A.1 · pp. 13–14

A.2 Data Construction Details

PEBench constructs cases through taxonomy-based scenario design, prompt diversification, reference acquisition, and automated checks followed by expert curation.

Stage 1: Taxonomy Design and Composition

  • Experts define nine capability categories and 35 tasks, then combine scenario attributes, a task, an enhancement setting, and target duration while discarding incompatible combinations.Coverage is monitored to preserve diversity without creating incoherent or infeasible requests.Source: Stage 1: Taxonomy Design and Composition · p. 14

Stage 2: Prompt Synthesis and Diversification

  • Gemini 3.1 Pro synthesizes requests from task, conditioning, scenario, and duration specifications, varying instruction detail and temporal structure from implicit shots to ordered events or scripts.The resulting inputs range from short concepts to detailed scripts and vary how much temporal planning is supplied to the enhancer.Source: Stage 2: Prompt Synthesis and… · p. 14

Stage 3: Reference Asset Acquisition

  • Reference assets come from screened public datasets and model-generated images; acquisition supports single, multiple, video-only, and mixed conditioning, with each asset assigned an explicit role.Public videos are screened for quality and motion prominence, segmented into semantically complete clips, and represented by keyframes when a static reference is needed.Source: Stage 3: Reference Asset Acquisition · p. 14

Stage 4: Automated Verification and Expert Curation

  • Automated checks assess task alignment, logical coherence, duration feasibility, and—when references are used—asset roles, script alignment, coverage, and cross-asset consistency.Ten domain experts then refine language, remove ambiguous or infeasible instructions, reject uninformative or mismatched references, and monitor balance, difficulty, and novelty.Source: Stage 4: Automated Verification and… · p. 14

A.3 Detailed Data Analysis

The appendix documents PEBench’s scenario diversity and details how its criteria assess semantic fidelity and internal consistency in enhanced prompts.

A.3 Detailed Data Analysis

  • Coverage spans nine capability categories, five subject categories, eight scene domains, seven T2V styles, 12 languages, and eight audio domains.The reported distributions include action as the largest capability category and English and Chinese as the most represented languages.Source: §A.3 · pp. 14–15
  • Prompt lengths and target durations vary; cinematic coverage spans shot scales, angles, lenses, and camera movements, while R2V varies asset modes and multi-reference configurations.R2V cases include image-only, video-only, and mixed image-video configurations, with multiple-image cases spanning two to seven references.Source: §A.3 · p. 15, Table 7 · p. 15

B.1 Evaluation Dimensions and Criteria

  • PEBench assesses 24 criteria grouped into Semantic Fidelity, Internal Consistency, and Directorial Creation, recording issue severity and evidence for applicable criteria.A criterion passes only when no issue is identified; optional-content criteria apply only when that content is present or explicitly requested.Source: §B.1 · p. 15

B.1.1 Semantic Fidelity Metrics

  • Semantic Fidelity checks preservation of the instruction and relevant visual evidence, including genre and style, subject attributes, required actions, event order, and narrative intent.Added detail is an issue if it omits, weakens, substitutes, or contradicts source requirements.Source: §B.1.1 · p. 16
  • Additional fidelity criteria cover on-screen text, dialogue and lyrics, audio requirements, camera constraints, lighting and color, and scene or spatial relations.The rubric includes exact wording and timing for visible text, preserves explicit camera intent, and allows compatible lighting or color elaboration without changing defining conditions.Source: §B.1.1 · p. 16

B.1.2 Internal Consistency Metrics

  • Internal Consistency evaluates whether the complete prompt is logically, spatially, temporally, physically, and audiovisually coherent, including subject continuity and action progression.Action and narrative logic checks prerequisites, subject-action bindings, reactions, resolution, and incompatible or dangling events.Source: §B.1.2 · p. 16
  • Its remaining criteria assess temporal and state continuity, spatial consistency, physical logic, and synchronization between visible events and audio.They flag issues such as infeasible timestamps, unexplained position swaps, violations of real-world or fictional rules, and misaligned audio-visual events.Source: §B.1.2 · pp. 16–17

B.1.3 Directorial Creation Metrics

This section examines whether enhanced prompts translate coherent requests into actionable cinematic and audiovisual plans, using prompt realizations of a scene in varied formats and levels of detail.

B.1.3 Directorial Creation Metrics

  • Directorial Creation assesses temporal planning, camera design, editing, lighting, narrative development, and whether additions stay focused on the core intent.Storyboard and timing completeness requires feasible event or shot boundaries, ordering, and durations; editing and transitions should support progression rather than present disconnected or unmotivated cuts.Source: §B.1.3 · p. 17
  • The remaining criteria assess content-appropriate sound integration, generation-relevant subject and performance details, and actionable visual-style specification.Sound design is judged by integration and visual timing, not by requiring every audio type; performance details can include appearance, expression, movement, interaction, gaze, and vocal delivery.Source: §B.1.3 · p. 17

B.2.1 Asset-Fact Extraction Output

  • The R2V AssetFacts example uses source-traceable facts to represent assets, their visual style and subjects, video segments, audio, and cross-asset relations.Its schema also includes visible scene and spatial relations, camera and composition, lighting and color, onscreen text, and uncertainty fields.Source: §B.2.1 · p. 17, R2V AssetFacts Output Format · pp. 17–20

B.2.2 Three-Dimension Evaluation Outputs

B.3 Representative Case

  • The representative example is a four-shot animated scene in which an older brother comforts a downcast nine-year-old boy in a garden.The provided input prompt begins with the brothers’ interaction and specifies a warm, uplifting atmosphere and a 3D family-film style.Source: §B.3 · p. 22, Input Prompt · p. 22

Input Prompt

  • The input prompt specifies a three-shot emotional arc with garden lighting, camera movement, piano, and ambient sounds.It progresses from the boy alone, through the brother’s approach and embrace, to the boy’s joyful reaction.Source: Input Prompt · p. 23

LingBot-Video-PE

  • LingBot-Video-PE organizes the scene into structured descriptions of characters, actions, timestamps, and camera details.Its description includes the younger boy’s shift from sadness to happiness and says the camera remains essentially stationary.Source: LingBot-Video-PE · p. 24

SCMaPR

  • SCMaPR gives a concise narrative of the brothers’ embrace and the younger boy’s change from melancholy to joy.A second version adds flowers, a bench, leaves, birds, and other garden details.Source: SCMaPR · p. 25

Prompt-A-Video-Cog

  • Prompt-A-Video-Cog describes the embrace as a warm, uplifting 3D family-film scene in a sunlit garden.Its language emphasizes the boy’s sorrow, the brother’s comforting gesture, and the younger boy’s joyful transformation.Source: Prompt-A-Video-Cog · p. 25

Prompt-A-Video-OS

  • Prompt-A-Video-OS offers a brief 15-second description of the garden scene, the comforting hug, and the boy’s change from sadness to joy.An alternate short prompt instead mentions a boy in a red shirt sitting on the ground and being hugged.Source: Prompt-A-Video-OS · p. 25

C.1 Implementation Details

The evaluation covers ten prompt-enhancement systems and an unenhanced baseline, with visual-input-capable systems additionally assessed on I2V-PE and R2V-PE. The evaluation uses cached modality-aware evidence and criterion-specific rubrics, with further analyses and human validation.

C.1 Implementation Details

  • Ten prompt-enhancement systems are evaluated alongside the original prompt as a no-enhancement baseline; visual-input-capable systems are also tested on I2V-PE and R2V-PE.Proprietary systems use their official interfaces, while released models use recommended inference settings.Source: §C.1 · p. 25
  • Cached modality-aware facts support rubric-based assessment of enhanced prompts across criterion pass rates and Semantic Fidelity, Internal Consistency, and Directorial Creation scores.GPT-5.4 extracts facts for image-only inputs and evaluates prompts; Qwen3.5-Omni-Plus processes cases with video references.Source: §C.1 · p. 26
  • Additional experiments examine temporal complexity, reference complexity, and output length, while human studies assess prompt-level alignment and downstream utility using Wan3.0 and MiniMax-H3.Wan3.0 is accessed through its online API with enhancement disabled; MiniMax-H3 runs locally on eight NVIDIA A100 GPUs.Source: §C.1 · p. 26

C.2 Length-Stratified Analysis

The length-stratified analysis finds that longer outputs do not consistently improve directorial guidance. The accompanying human-validation protocol, interface, and ethical release practices define how judgments are collected and materials are handled.

C.2 Length-Stratified Analysis

  • Across all three tasks, the longest output quartile underperforms the shortest by 5.19 to 10.09 DC points, so longer outputs do not consistently provide better directorial guidance.Directorial Creation rewards concrete, relevant, executable specifications rather than verbosity or repeated template language.Source: Table 8 · p. 26, §C.2 · p. 26

D.1 Pairwise Annotation Protocol

  • Sixty trained experts make dimension-specific pairwise judgments of enhanced prompts and generated videos, whose rankings are compared with automatic scores using Spearman correlation.The study uses Bradley-Terry aggregation and excludes annotators with significant left- or right-position bias before fitting rankings and computing correlations.Source: §D.1 · pp. 26–27

D.2 Annotation Interface

  • The interface presents the source prompt, duration, and applicable references alongside anonymized prompt or video pairs, with independent rating rows for three evaluation dimensions.Candidate positions are randomized; video comparisons support synchronized or individual playback and replay.Source: §D.2 · p. 27

E Ethics, Privacy, and Licensing

  • Curation removes personal information and sensitive material, tracks source and license metadata, and releases only redistributable assets while excluding validation videos from the benchmark release.Restricted materials are represented by source identifiers.Source: E Ethics, Privacy, and Licensing · p. 27

F Limitations

PEBench’s 1,100 cases provide broad coverage, but cannot represent the full range of rapidly evolving video-generation requests. I2V-PE and R2V-PE comparisons also cover fewer baselines because few existing methods support multimodal prompt enhancement.

F Limitations

  • The 1,100-case benchmark cannot represent the full range of rapidly evolving video-generation requests despite coverage of three settings, 35 task types, and diverse conditioning patterns.The authors plan to expand domains, languages, reference configurations, and emerging control formats.Source: F Limitations · p. 27
  • I2V-PE and R2V-PE evaluations include fewer baselines than T2V-PE because few existing methods support multimodal prompt enhancement and accessible interfaces are scarce.The authors plan to update PEBench as more multimodal methods become available.Source: F Limitations · p. 27
Loading 2610.11736v1…