PaperlayerView exact revision on arXiv

Source-linked AI summary

KeyFrame-Compass: Towards Comprehensive Evaluation of Keyframe-Conditioned Video Generation

Yuqi Tang, Tengfei Liu, Yizheng Lai, Yuran Wang, Yang Shi, Wanshun Su, Zhuoran Zhang, Qixun Wang, Xiaohan Zhang, Xinlei Yu, Xuehai Bai, Xuanyu Zhu, Bohan Zeng, Bozhou Li, Shujie Li, Yifan Dai, Yujie Wei, Shixuan Liu, Haotian Wang, Jialu Chen, Yuanxing Zhang

arXiv:2607.14202v1cs.CV

TL;DR

Existing video benchmarks do not assess whether a complete prescribed keyframe sequence is reproduced with the correct appearance, order, and timing. KeyFrame-Compass evaluates keyframe execution and overall video quality, and experiments reveal a trade-off between faithful keyframe execution and natural video synthesis, with controllability worsening as constraints become denser.

  • Problem

    Existing benchmarks do not measure whether a complete prescribed keyframe sequence is reproduced with the correct appearance, order, and timing.

    Source: §1 · p. 2

  • Method

    KeyFrame-Compass evaluates 386 cases across controlled generation settings using six keyframe-execution metrics and evidence-grounded video-quality judgments augmented with specialized perception models.

    Source: §1 · p. 2

  • Results

    Evaluated systems show a trade-off between keyframe fidelity and natural video synthesis, and controllability deteriorates as keyframe constraints become denser.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Keyframe fidelity, temporal organization, and general generation quality are noninterchangeable capabilities, so rankings differ depending on the evaluation axis.

    Source: §4.2 · pp. 9–10

    For LTX-2.3, the principal remaining limitation is synthesizing plausible transitions between keyframes, especially across large scene, viewpoint, or shot-scale changes.

    Source: §4.3 · p. 10

Abstract

from arXiv · show

Video generation increasingly relies on keyframe-based workflows, where creators specify a sequence of reference images to guide generation. Although recent models support multi-keyframe conditioning, it remains unclear whether they can faithfully reproduce the prescribed keyframes while maintaining overall video quality. We present KeyFrame-Compass, the first comprehensive benchmark for evaluating keyframe-conditioned video generation. The benchmark contains 386 carefully curated samples spanning three application domains, two video structures, two prompt granularities, two conditioning formats, and four keyframe densities, enabling controlled analysis under diverse generation settings. We further introduce an automated evaluation framework that jointly measures keyframe execution and overall video quality. Specifically, we decompose keyframe execution into six complementary metrics covering presence, fidelity, temporal ordering, localization, persistence, and uniqueness, while assessing overall video quality through evidence-grounded MLLM judgments augmented with specialized perception models. Experiments on nine representative video generation systems reveal several fundamental limitations. Current models exhibit a clear trade-off between faithful keyframe execution and natural video synthesis. Their performance further degrades as keyframe constraints become denser and most open-source models also fail to interpret storyboard-grid inputs as temporally ordered keyframe sequences.

1. Introduction

KeyFrame-Compass benchmarks multi-keyframe-conditioned video generation with controlled task settings and separate measures of keyframe execution and video quality. Evaluations reveal trade-offs between faithful keyframe execution and natural synthesis, with controllability worsening as constraints grow denser.

Motivation

  • Existing video and image-conditioned benchmarks do not measure whether a complete keyframe sequence is reproduced with correct appearance, order, and timing.Keyframe-conditioned generation must preserve ordered reference images, place them at the right times, and connect them with coherent motion; failures include omission, inaccurate reproduction, mistiming, reordering, and implausible transitions.Source: §1 · p. 2

Benchmark design

  • KeyFrame-Compass comprises 386 cases across three application domains and five controlled factors, including video structure, keyframe count, prompt specificity, and conditioning interface.It uses curated keyframes spanning daily capture, product visualization, and cinematic narrative, with realistic and stylized content; keyframe counts are 3, 6, 9, or 12.Source: §1 · p. 2

Evaluation framework

  • The evaluation separates six keyframe-response metrics from general video-quality measures combining MLLM judgments and specialized perception models.Keyframe response covers presence, fidelity, ordering, timing, persistence, and uniqueness; Figure 2 describes a staged matching pipeline for locating keyframes in exact temporal windows and four general-quality groups.Source: §1 · p. 2, Figure 2 · p. 3

Results

  • Models trade off keyframe fidelity against natural synthesis, controllability declines with denser constraints, and the strongest open-source model trails the leading proprietary system by 0.153 overall.Close adherence can produce abrupt or implausible transitions, while looser use of keyframes can drift from their visual content; no evaluated system performs best on both axes.Source: §1 · p. 2

2. Related Work

Prior work has expanded image-conditioned video generation and benchmark coverage, but existing evaluations do not diagnose whether ordered keyframes are reproduced at intended temporal locations. KeyFrame-Compass addresses this gap with diagnostic evaluation of keyframe response and overall video quality.

2.1. Image-Conditioned Video Generation

  • Recent image-conditioned methods use storyboards and predefined keyframes, arbitrary frame guidance, and interleaved multimodal inputs, yet lack unified ordered-keyframe evaluation.These methods offer stronger control over event states, shot composition, pacing, and cross-shot consistency [48], while evaluations focus on generic quality, reference fidelity, transition smoothness, or consistency.Source: §2.1 · p. 3

2.2. Benchmarks for Video Generation

  • VBench [16], EvalCrafter [22], TC-Bench, and VBench-2.0 [47] assess quality, prompt following, temporal constraints, compositionality, and physical plausibility.Other benchmarks assess audio-video coordination, long-form coherence, or script-conditioned generation, including VABench, T2AV-Compass [5], AVGen-Bench, LongAV-Compass, and MSVBench.Source: §2.2 · p. 3
  • Existing benchmarks do not assess ordered keyframe reproduction at intended temporal locations with coherent transitions; KeyFrame-Compass combines keyframe-response evaluation with comprehensive video-quality assessment.AIGCBench [8] and UI2V-Bench focus on single-image conditioning, while VideoCanvasBench and ViStoryBench address spatiotemporal completion or story visualization.Source: §2.2 · p. 3

3. KeyFrame-Compass

KeyFrame-Compass benchmarks whether video generators realize ordered visual keyframes while producing coherent video, using a curated dataset, a multicomponent evaluation framework, and diverse task settings.

3.1. Benchmark Design

  • Samples test ordered keyframe realization across two input formats, two prompt granularities, two video structures, four keyframe counts, and three application domains.Multi-shot keyframes receive first, last, or representative shot positions; one-take keyframes receive target timestamps, with segments specifying intervening evolution.Source: §3.1 · pp. 3–4

3.2. Data Construction

  • Construction turns narrative sources into structured scene specifications, generated keyframes, and adapted video prompts, with multimodal checks, human review, and safety screening.Sources include stories from ViStoryBench[50], VIST, ROCStories, and selected real videos; keyframes are generated with GPT-Image-2 and Nano Banana Pro, while Gemini 3.1 Pro adapts video prompts.Source: §3.2 · pp. 4–5, Figure 3 · p. 5

3.3. Evaluation Metrics

  • Six keyframe-response metrics diagnose presence, fidelity, order, timing, persistence, and uniqueness, alongside general video-quality, coherence, instruction-adherence, and audio-visual assessments.General-quality judgments use checklist-based evidence verification, with specialized perception-model outputs incorporated where applicable.Source: §3.3 · pp. 5, 7

3.4. Statistics

  • The benchmark contains 386 cases across three video structure-duration settings and three application domains, with keyframe counts from 3 to 12.It includes 75, 112, 110, and 89 samples with 3, 6, 9, and 12 keyframes, respectively; stories average 3.73 unique characters.Source: §3.4 · pp. 8–9, Table 2 · p. 9

4. Experiment

KeyFrame-Compass evaluates keyframe execution alongside overall video quality, revealing trade-offs in model rankings and persistent limitations in transitions, dense constraints, and storyboard-grid comprehension.

4.1. Experimental Settings

  • The evaluation covers nine models, using single-pass workflows for videos of 10 seconds or shorter and agent-mode multi-pass workflows for longer videos.Eligible models share test cases within duration splits and prompt variants; multi-image models receive image lists, while single-image models receive storyboard grids. Outputs are generally 720P, with daVinci-MagiHuman-1080p-I2V at 1080P.Source: §4.1 · p. 9

4.2. Main Results

  • Seedance 2.0 ranks first overall at 0.807 on the 115-sample joint audio-video leaderboard, exceeding Gemini-Omni-Flash by 0.063.LTX-2.3 leads in Keyframe Fidelity (0.855) and Temporal Organization (0.899) but ranks fourth overall (0.659), while Gemini-Omni-Flash leads in Video Quality (0.861) and AV Coordination (0.640). The rank reversals show that keyframe fidelity, temporal organization, and general video quality are noninterchangeable; Overall averages the two prompt-mode scores after joint keyframe-control aggregation.Source: §4.2 · pp. 9–10, Table 3 · p. 10

4.3. Analysis and Insights

  • LTX-2.3 achieves the strongest open-source performance and highest keyframe response scores, but its weaker transition dynamics remain a principal limitation.Large scene, viewpoint, or shot-scale changes can yield slideshow-like transitions, while rapid motion can produce melting or dissolving artifacts; its minimal-mode DVQ is 0.453 and PR is 0.417.Source: §4.3 · p. 10, Figure 6 · p. 12
  • Gemini-Omni-Flash often preserves semantic shot instructions while reconstructing scene details, whereas Seedance 2.0 more strongly anchors appearance to keyframes.Gemini ranks among the top models on VMA but has relatively low KFS; Seedance’s stricter anchoring can limit context-appropriate refinement or completion.Source: §4.3 · pp. 10–12
  • For all four proprietary models, segment-specific prompts lower VMA by 0.060–0.260 versus minimal prompts, and instruction scores fall from 0.849 at 3 keyframes to 0.756 for 9&12.The decline is driven primarily by VMA, while AMA remains nearly unchanged across keyframe-count groups.Source: §4.3 · p. 12, Table 5 · p. 12
  • Four open-source models using storyboard grids have Hit Rates below 0.28 and often reproduce the grid as a near-static video rather than ordered shots.They also produce unrelated visual artifacts, and daVinci sometimes reads prompt text aloud or applies one shot’s audio requirements to another; these comprehension failures are absent in proprietary systems.Source: Figure 6 · p. 12, §4.3 · p. 13

4.4. Human Alignment

  • Human–automatic model-ranking correlations range from Spearman’s ρs = 0.70 to 1.00 across five models.The study used 60 segment-level cases and 600 pairwise comparisons, with each comparison judged by three experts.Source: §4.4 · p. 13, Table 6 · p. 13

5. Conclusion

KeyFrame-Compass is a 386-sample benchmark and automated evaluation framework for keyframe-conditioned video generation. Experiments expose limitations in following dense constraints and producing coherent transitions, storyboard-grid inputs, and one-take videos.

5. Conclusion

  • Current models struggle with fine-grained instructions under dense keyframe constraints, coherent transitions, storyboard-grid inputs, and one-take video generation.The passage notes that open-source models particularly struggle to decompose storyboard-grid inputs.Source: §5 · p. 13
  • The benchmark comprises 386 samples across four keyframe densities, two video structures, two prompt settings, two input formats, and three application domains.This scope enables systematic evaluation across diverse generation scenarios.Source: §5 · p. 13
  • The automated evaluation framework combines keyframe-response metrics with general video-quality metrics using vision-language judges and specialized perception models.Keyframe-response metrics assess reproduction at the intended temporal position and duration.Source: §5 · p. 13
  • KeyFrame-Compass aims to standardize diagnosis of these limitations and facilitate progress in keyframe-conditioned video generation.Source: §5 · p. 13

Supplementary Material · A. Case Study

The case study presents representative generated videos from open-source and proprietary models to show their output quality and generation preferences.

A. Case Study

  • Representative videos from open-source and proprietary models illustrate their output quality and generation preferences, and examples are read in row-major order.Source: A. Case Study · p. 16

A.1. Performance of Proprietary Models

Proprietary models perform strongly overall but favor different balances of keyframe fidelity, coherence, and instruction alignment; Wan 2.7 generally interprets storyboard grids into complete stories despite format-related output differences.

A.1. Performance of Proprietary Models

  • Seedance 2.0 shows the highest fidelity to input keyframes, preserving visual details and temporal anchors more accurately.Source: §A.1 · p. 16
  • Gemini-Omni-Flash may not faithfully reproduce keyframes, but it generates coherent, fluent videos with complete story progression and good instruction alignment.Source: §A.1 · p. 16
  • Kling-3.0-Omni balances keyframe correspondence and narrative coherence while maintaining relatively stable quality.Source: §A.1 · p. 16
  • Wan 2.7 generally interprets storyboard grids into complete stories, although its single-image input support yields different output resolution and aspect ratio.Source: §A.1 · p. 16

A.2. Performance of Open-Source Models

Among open-source models, LTX 2.3 recovers most storyboard-grid keyframes, but its transitions remain rigid and other models often produce collage-like outputs.

A.2. Performance of Open-Source Models

  • LTX 2.3 performs strongest among open-source models, reproducing most storyboard-grid keyframes but with rigid transitions, while other models often produce static collages instead of coherent videos.Its transitions often appear as hard cuts or slide-like changes rather than continuous motion.Source: §A.2 · p. 16

B. Calibration and Computation of Keyframe Similarity

The section calibrates semantic keyframe matching on held-out validation pairs, defines pixel-score normalization and fusion, and finds that KFS is stable across tested pixel-score settings.

B. Calibration and Computation of Keyframe Similarity

  • For each input keyframe, the method selects the response frame with the highest cosine similarity between L2-normalized DINOv3 features.The validation labels used a relaxed semantic criterion: at least one salient subject, object, or event participant had to be shared.Source: B. Calibration and Computation of… · p. 16
  • The equal-error threshold was γ = 0.7837 (FPR 5.3%, FNR 5.4%); deployed γ⋆ = 0.80 yielded validation FPR 5.3% and FNR 6.4%.Threshold selection used 1,027 labeled pairs after excluding three uncertain cases from 1,030 inspectable pairs, and the threshold was fixed before test evaluation.Source: B. Calibration and Computation of… · p. 16
  • For matched keyframes, the pixel score uses PSNR and SSIM with lower anchors τp = 12.0 dB and τs = 0.65, and saturation values of 50 dB and 1.0.These parameters scale the continuous pixel score but do not determine whether a pair is semantically matched.Source: B. Calibration and Computation of… · pp. 16–17
  • The final score combines pixel and semantic similarities for matched keyframes, while unmatched keyframes contribute zero and remain in the denominator.The denominator is the total number of input keyframes.Source: B. Calibration and Computation of… · p. 17
  • Across tested parameter ranges, mean KFS changed by at most 0.0244, and minimum Spearman rank correlation with the default setting was 0.997.The sensitivity analysis covered 436 outputs and 2,892 keyframes from 52 minimal-prompt and 57 specific-prompt samples.Source: B. Calibration and Computation of… · pp. 17–18

C. MLLM Judge Sampling and Stability

The evaluation uses 8-FPS video sampling for MLLM-judged metrics, with scores changing only slightly at lower rates and repeated-run aggregate scores showing low variation. The section also specifies how repeated-run metrics are aggregated across models and prompt modes.

C. MLLM Judge Sampling and Stability

  • Scores vary only slightly at 4 and 2 FPS, so all reported MLLM-judged results use 8-FPS sampling.Videos are sampled uniformly, with frames supplied chronologically; the sampling-rate comparison keeps the judge, prompt, and scoring procedure fixed.Source: C. MLLM Judge Sampling and Stability · p. 18
  • Low standard deviations across five evaluation runs indicate that aggregated MLLM scores are stable across independent judge invocations.Runs use identical inputs and judge configurations on the 115-sample common set; Table 7 reports group-score macro-averages across six models and two prompt modes. Visual Quality averages SVQ and DVQ; Spatiotemporal Coherence averages AC, SOC, and PR; Instruction Adherence averages applicable VMA and AMA scores, omitting AMA for minimal prompts.Source: C. MLLM Judge Sampling and Stability · pp. 18–19

D. Human Alignment Protocol

The human alignment protocol compares five models on a shared 60-case subset using blinded expert judgments across four dimensions. It defines how human preferences and automatic metrics are aggregated and how agreement and rank correlations are evaluated.

Evaluation protocol

  • The evaluation uses 60 cases—10 one-take and 50 multi-shot—with 20, 20, 10, and 10 cases at 3, 6, 9, and 12 keyframes, respectively.All five evaluated models have complete video outputs and automatic metric results for the subset.Source: D. Human Alignment Protocol · p. 19
  • Across 600 model-pair comparisons, three of twelve domain experts independently choose Video A, Video B, or Tie while model identities and automatic scores are hidden.Annotators see the input keyframes and segment-specific prompt; video order is randomized left–right.Source: D. Human Alignment Protocol · p. 19

Human rubric

  • The four human dimensions are Video Quality, Spatiotemporal Coherence, Instruction Adherence, and Audio-Visual Coordination, with ties used when neither video is consistently preferable.Instruction Adherence covers only prompt-specified requirements, whereas Audio-Visual Coordination judges intrinsic audio–video semantic and temporal agreement independently of the prompt.Source: D. Human Alignment Protocol · p. 19

Aggregation and alignment

  • Human win rates average preference scores across each model’s four opponents, 60 cases, and three annotators, and the overall human score averages the four dimension-level rates.A selected model receives 1 and its opponent 0; ties contribute 0.5 to each, yielding 720 judgments per model per dimension.Source: D. Human Alignment Protocol · p. 20
  • Pairwise agreement counts decisive human majorities against per-case automatic-score differences, while model-level Spearman correlations use all annotations and exact two-sided tests over 120 permutations.Agreement excludes majority-tie and one-A/one-B/one-Tie outcomes, reports decisive-item coverage, and uses 10,000 case-clustered bootstrap replicates for 95% confidence intervals; Krippendorff’s α is summarized across ten model pairs.Source: D. Human Alignment Protocol · pp. 20–22

E. Prompt Templates

This section describes KeyFrame-Compass’s prompt templates and evaluation contracts for video segmentation, metric-specific checklist generation, and checklist scoring. The templates define scoped, evidence-grounded checks and scoring guidance across prompt types, modalities, and evaluation metrics.

E. Prompt Templates

  • The section’s prompt-template scope also includes the segmentation contract and grouped judge prompt with scoring rubrics, alongside generation and checklist prompts.Source: E. Prompt Templates · p. 23

E.1. Prompt Templates for Video Generation

  • Segment-specific prompts organize multi-shot videos by shot and single-shot videos by keyframe-to-keyframe segment, with single-shot prompts also specifying global camera motion.For multi-shot videos, each shot is the minimum unit; for single-shot videos, keyframes act as temporal anchors and content between adjacent keyframes is specified segment by segment.Source: §E.1 · p. 24
  • The single-shot template includes global camera motion, treats keyframes as ordered anchors, and requests smooth continuity while preserving each keyframe’s appearance.Source: §E.1 · pp. 24–25
  • Minimal prompts provide keyframe order, a brief synopsis, target video structure, and duration, leaving the model to infer events and transitions from the visual sequence.Source: §E.1 · p. 25

E.2. Prompts for Video Segmentation

  • Segmentation follows the generated video rather than the requested structure: the evaluator first identifies actual edit boundaries, then assigns keyframes to the fixed shots.A continuous take remains one shot; action phases, camera movement, and state changes do not create new shots.Source: §E.2 · pp. 25–26
  • Keyframe assignment must not alter shot boundaries; keyframes with no visible response are left unassigned rather than forced into a segment.Source: §E.2 · p. 26

E.3. Prompts for Checklist Generation

  • Checklist generation runs independently for each metric, using a filtered sample specification, metric-specific construction procedure, and response shape to produce inspection lenses without video access or scoring.Source: §E.3 · p. 26
  • Checklist items use five semantic fields, while benchmark code adds schema metadata; the judge applies metric rubrics rather than generated item-level pass/fail rules.Source: §E.3 · p. 27
  • The prompt builds atomic observable checks from sample specs, keyframes, shots, narrowly interpreted hints, and metric procedures, anchoring claims to relevant keyframes or leaving unanchored items empty.Source: §E.3 · p. 27
  • Static Visual Quality evaluates frame-level clarity, artifacts, color and lighting aesthetics, and style stability, excluding anatomy, physical behavior, prompt adherence, and identity or attribute drift.Source: §E.3 · p. 28
  • Static checks favor 3–6 single-subdimension items, using shot or segment scope for localized quality and full-video scope for style consistency or pervasive issues.Source: §E.3 · p. 28
  • Dynamic quality assigns temporal artifacts to distinct owners: jumps or replacements, stagnant intervals, in-place morphing, recurring within-shot flicker, or boundary glitches.Source: §E.3 · p. 28
  • Dynamic checks use full-video scope for recurring artifacts, cross-shot scope for boundary glitches, localized segments when appropriate, and one artifact criterion per item.Source: §E.3 · pp. 28–29
  • Attribute Consistency classifies recurring characters, animals, and props from the sample context, then creates identity or attribute persistence checks anchored to relevant keyframes.Source: §E.3 · p. 29
  • Attribute-consistency checks track recurring entities’ identity and attributes across relevant spans, allowing prompt-requested changes but checking whether new states persist.Items prioritize entities visible in keyframes or central to the prompt and link attributes to the keyframes that anchor them.Source: §E.3 · p. 29
  • Checklist construction uses category-specific item counts and evidence scopes, names the entities or property checked, and grounds questions in visible evidence across a video or major segment.Attribute, spatial, and physical procedures prefer 4–8, 2–5, and 3–5 items respectively; physical checks default to full-video scope, while spatial checks use cross-shot or full-video scope as appropriate.Source: §E.3 · pp. 29–30
  • Spatial-orientation checks assess visible relative positions and scale proportions, treating changes as failures only when motion, viewpoint shifts, or scene transitions do not explain them.Use visible media and filtered hints, distinguishing camera-induced layout changes from actual subject or object drift.Source: §E.3 · p. 29
  • Physical-rationality checks cover interaction, biological anatomy, non-biological structure, scene-level common sense, and contextually important text readability without duplicating failures.Items target visible plausibility failures across the video, including issues not predictable from the prompt or keyframes.Source: §E.3 · p. 30
  • Physical-check boundaries assign failures to specific dimensions, reserving scene-level common sense for behavior and world logic rather than contact, anatomy, object structure, or text legibility.Text readability applies when important visible text becomes nonsensical or unstable, not merely small, blurred, compressed, or poorly exposed.Source: §E.3 · p. 30
  • Video-modality checks assess requested camera behavior, shot count or continuous-take structure, narrative rhythm and focus, and required subjects, props, actions, or scenes in intended segments.They evaluate prompt/spec adherence rather than internal visual quality, spatial consistency, or physical plausibility.Source: §E.3 · pp. 30–31
  • Checklist-generation prompts use single-requirement items and scope them to shots, cross-shot transitions, or the full video as appropriate.For full-video story-focus checks, they suggest asking whether the sequence preserves the intended focus rather than spending most of its time on unrelated actions.Source: §E.3 · p. 31
  • Audio-adherence prompts check requested sounds, music, dialogue, ambience, and audio style or emotion, while excluding audio-visual synchronization.They use audio_track evidence when audio is requested and mark the metric not applicable when no audio requirement exists.Source: §E.3 · p. 31
  • Minimal-prompt video-adherence prompts ground checks in keyframes and the story summary, covering visible subjects, shot structure, and implied events without inventing absent details.They distinguish multi-shot samples from continuous one-take samples and use adjacent-keyframe differences to infer expected rhythm.Source: §E.3 · pp. 31–32
  • Attribute-consistency prompts identify visible characters, animals, and props from keyframes, then check recurring or central subjects using one identity cue per item.Each check uses the matching entity-type dimension and cites every keyframe that visibly anchors the attribute.Source: §E.3 · p. 32
  • Spatial-orientation prompts check keyframe-grounded relative positions and plausible scale, treating relation changes as failures only when movement, viewpoint, or transitions do not explain them.Checks use only clearly visible relations and name the subjects and relation being evaluated.Source: §E.3 · p. 32

E.4. Prompts for Grouped Judge

  • The grouped judge receives the generated video, input keyframes, filtered evaluation context, and precomputed checklists, resolves keyframe references, and returns separate results for each metric.Related metrics are evaluated together, with timestamped observable evidence required.Source: §E.4 · p. 32
  • The judge scores every checklist item unless evidence is genuinely insufficient, and the metric-level score is the arithmetic mean of valid item results.The parser recomputes this mean, so score consistency matters.Source: §E.4 · p. 33
  • Each checklist score must rest on observable evidence, with video or audio evidence tied to timestamps and described concretely enough to support pass, partial, or fail.Generic judgments without timestamped observations are insufficient; when full video is attached, the template prefers time ranges over frame IDs.Source: §E.4 · p. 33
  • The grouped judge scores each checklist item on a normalized 0–1 scale, and the parser recomputes each metric score as the arithmetic mean of valid item scores.Each result copies the stable checklist identifier and records timestamped evidence; auxiliary tool evidence may also be cited.Source: §E.4 · p. 33
  • The response format requires strict JSON with one object per checklist metric and a result for each item, including its identifier, score, evidence, and support.The template also allows timestamped observations and auxiliary tool-evidence IDs.Source: §E.4 · p. 33
  • Each metric receives its own rubric section defining evaluation ownership and normalized score anchors without adding checklist fields.Source: §E.4 · p. 33
  • The static visual rubric judges frame-level clarity, artifacts, color and lighting, and style consistency within the checklist item’s scope.Source: §E.4 · p. 33
  • The dynamic visual rubric assesses temporal defects such as discontinuities, stutter or repetition, implausible morphing, and flicker or instability.Source: §E.4 · p. 34
  • The attribute rubric checks whether named entities and identity cues remain recognizable across the scoped evidence, allowing for prompt-requested changes.Its cues include character, animal, and prop attributes; scores range from stable matches to missing, replaced, or persistently wrong cues.Source: §E.4 · p. 34
  • The spatial rubric evaluates relations, ordering, and proportions while accounting for visible movement, scene changes, and viewpoint.MonST3R sensor context, when present, is auxiliary camera-motion context and does not provide object masks, boxes, depth, or pose evidence.Source: §E.4 · p. 34
  • The physical-rationality rubric assigns visible plausibility failures to dimensions covering interactions, anatomy, structural loss, materials, text readability, and scene-level common sense.Source: §E.4 · pp. 34–35
  • The physical-behavior guidance assigns 0.25 to repeated, severe, or distracting errors and 0.0 when impossible or contradictory behavior dominates or breaks scene logic.The rubric concerns whether scoped physical behavior remains understandable and acceptable.Source: §E.4 · p. 35
  • Video adherence is judged against the named prompt/spec requirement; for minimal prompts, only expectations inferable from keyframes or story_summary count.When present, MonST3R sensor_context is auxiliary camera-motion context; the visible video remains primary.Source: §E.4 · p. 35
  • Video dimension cues cover camera motion and framing, shot structure, narrative rhythm, story focus, and the presence of required characters, props, actions, and scenes.One-take means a single continuous shot with no cuts; key events should receive appropriate visual time, and the video should stay focused on the intended story.Source: §E.4 · p. 35
  • Video adherence scores range from 1.0 for clearly present, correctly timed and placed content to 0.0 for missing, contradicted, misplaced, or unrelated content.Scores of 0.75, 0.5, and 0.25 represent progressively more substantial issues, from minor problems to weak or ambiguously localized adherence.Source: §E.4 · p. 35
  • Audio adherence checks apply to the named requirement, but an item marked not_applicable must not prompt an invented audio expectation.Cues assess requested sounds and their match to visual events, audio style and emotion, and synchronization with visible causes or actions when implied.Source: §E.4 · p. 35
  • Audio scores range from 1.0 for clearly satisfied, matched, appropriate, and synchronized requirements to 0.0 for missing, unusable, contradicted, or unrelated sound.Intermediate scores capture minor mismatches (0.75), partial support with an important property missing (0.5), and weak, generic, faint, or poorly synchronized audio (0.25).Source: §E.4 · p. 35

Related papers

Loading 2607.14202v1…