PaperlayerView exact revision on arXiv

Source-linked AI summary

KeyFrame-Compass: Towards Comprehensive Evaluation of Keyframe-Conditioned Video Generation

Yuqi Tang, Tengfei Liu, Yizheng Lai, Yuran Wang, Yang Shi, Wanshun Su, Zhuoran Zhang, Qixun Wang, Xiaohan Zhang, Xinlei Yu, Xuehai Bai, Xuanyu Zhu, Bohan Zeng, Bozhou Li, Shujie Li, Yifan Dai, Yujie Wei, Shixuan Liu, Haotian Wang, Jialu Chen, Yuanxing Zhang

arXiv:2607.14202v1cs.CV

TL;DR

Existing benchmarks do not assess whether multi-keyframe video generators reproduce complete keyframe sequences with correct appearance, order, and timing. KeyFrame-Compass addresses this gap with a controlled benchmark and joint evaluation framework, finding a persistent trade-off between keyframe fidelity and natural video synthesis as constraints become denser.

  • Problem

    Existing benchmarks do not measure whether complete keyframe sequences are reproduced with correct appearance, order, and timing.

  • Method

    KeyFrame-Compass evaluates 386 cases across controlled task factors using six keyframe-execution metrics and automated overall-quality assessment.

  • Results

    Models show a persistent trade-off between keyframe fidelity and natural video synthesis, with controllability deteriorating as keyframe constraints become denser.

  • Takeaways & Limitations

    Keyframe adherence and general video quality are noninterchangeable capabilities, so evaluating either dimension alone can produce different model rankings.

  • Takeaways & Limitations

    For LTX-2.3, the principal remaining limitation is synthesizing plausible transitions between visual anchors, especially across large scene or viewpoint changes.

Abstract

from arXiv · show

Video generation increasingly relies on keyframe-based workflows, where creators specify a sequence of reference images to guide generation. Although recent models support multi-keyframe conditioning, it remains unclear whether they can faithfully reproduce the prescribed keyframes while maintaining overall video quality. We present KeyFrame-Compass, the first comprehensive benchmark for evaluating keyframe-conditioned video generation. The benchmark contains 386 carefully curated samples spanning three application domains, two video structures, two prompt granularities, two conditioning formats, and four keyframe densities, enabling controlled analysis under diverse generation settings. We further introduce an automated evaluation framework that jointly measures keyframe execution and overall video quality. Specifically, we decompose keyframe execution into six complementary metrics covering presence, fidelity, temporal ordering, localization, persistence, and uniqueness, while assessing overall video quality through evidence-grounded MLLM judgments augmented with specialized perception models. Experiments on nine representative video generation systems reveal several fundamental limitations. Current models exhibit a clear trade-off between faithful keyframe execution and natural video synthesis. Their performance further degrades as keyframe constraints become denser and most open-source models also fail to interpret storyboard-grid inputs as temporally ordered keyframe sequences.

1. Introduction

KeyFrame-Compass addresses the lack of benchmarks for multi-keyframe-conditioned video generation with a controlled benchmark and evaluation framework. Experiments reveal a persistent trade-off between faithful keyframe execution and natural video synthesis, alongside systematic performance patterns and failure modes.

  • Motivation: Existing benchmarks assess general video quality or single-image conditioning, but do not measure whether generated videos execute ordered multi-keyframe plans.This gap motivates a dedicated diagnostic benchmark for multi-keyframe-conditioned video generation.
  • Benchmark: KeyFrame-Compass evaluates ordered keyframes and text prompts across daily capture, product visualization, and cinema domains while requiring coherent transitions.The benchmark is designed to test both reproduction of the specified visual sequence and transitions between adjacent keyframes.
  • Benchmark: 386 evaluation cases span video structure, keyframe count, prompt specificity, conditioning interface, and application domain.Keyframe counts are 3, 6, 9, or 12; interfaces include multi-image lists and storyboard grids.
  • Evaluation: The framework jointly measures keyframe-plan execution through presence, fidelity, ordering, timing, persistence, and uniqueness, and evaluates general video quality.It combines multimodal large language model judgments with specialized perception models for quality assessment.
  • Findings: Experiments reveal a persistent trade-off: models closely following keyframes may produce abrupt or implausible transitions, whereas natural synthesis can weaken keyframe fidelity.The evaluation covers representative systems under both prompt control levels, with videos up to 10s generated in one pass and longer videos produced through agent-mode workflows when available.

2. Related Work

Prior work advances image-conditioned video generation from single-image animation toward richer keyframe, storyboard, arbitrary-frame, and multimodal controls. Existing benchmarks largely assess output quality and related capabilities, but do not directly evaluate faithful reproduction of ordered keyframes at intended temporal locations.

  • Image-Conditioned Video Generation: Recent image-conditioned video methods extend single-image animation with storyboards, predefined keyframes, arbitrary frame guidance, and interleaved multimodal inputs.These controls target key event states, shot composition, narrative pacing, and cross-shot consistency.
  • Video Generation Benchmarks: General video benchmarks evaluate perceptual quality, prompt following, temporal constraints, compositionality, physical plausibility, audio-video coordination, long-form coherence, and script-conditioned generation.Examples include VBench, EvalCrafter, TC-Bench, VBench-2.0, VABench, T2AV-Compass, AVGen-Bench, LongAV-Compass, and MSVBench.
  • Benchmarking Gaps: Existing benchmarks assess final outputs rather than faithful reproduction of ordered keyframes at intended temporal locations with coherent transitions.AIGCBench and UI2V-Bench mainly evaluate single-image conditioning, whereas VideoCanvasBench and ViStoryBench extend evaluation to arbitrary spatiotemporal completion and story-based video generation.

3. KeyFrame-Compass

KeyFrame-Compass benchmarks whether video generation systems reproduce ordered visual keyframes while synthesizing intervening motion, transitions, and events. It stratifies tasks across controlled generation settings and evaluates both keyframe execution and overall video quality.

  • Task definition: Each task supplies keyframe images and a text prompt, requiring the generated video to realize specified visual states in temporal order and synthesize transitions between anchors.Multi-shot keyframes occupy first, last, or representative positions within assigned shots; one-take keyframes target timestamps along a continuous trajectory.
  • Benchmark design: The benchmark stratifies samples by input format, prompt control, video structure, keyframe count, and application domain for fine-grained evaluation.Keyframes are provided either as ordered individual images or as a storyboard grid containing the same content and ordering.
  • Data construction: Samples are constructed from narrative datasets and selected real videos, converted into structured story summaries and scene specifications, then filtered through multimodal checks, human review, and safety screening.Video-oriented prompts preserve subject identity, visual states, and spatial relations during adaptation.
  • Evaluation framework: Evaluation combines keyframe response metrics for fidelity, timing, and temporal structure with general quality metrics for visual quality, coherence, instruction adherence, and audio–visual alignment.The six keyframe metrics jointly assess presence, fidelity, ordering, timing, persistence, and uniqueness, while checklist-based judgments use GPT-5.5 and Gemini 3.1 Pro.

4. Experiment

The experiment evaluates nine video generation systems under duration-specific, prompt-controlled conditions using a unified pipeline. Results reveal trade-offs between overall quality and keyframe execution, weaker transition synthesis, declining instruction adherence with denser constraints, and poor storyboard-grid comprehension among most open-source models.

  • Experimental Setup: Nine representative systems are evaluated on shared test cases under both prompt variants, with short videos tested in single-pass mode and long videos using agent-mode workflows when supported.Short-video evaluation includes four proprietary models, while long-video evaluation is limited to Kling-3.0-Omni and Seedance 2.0.
  • Leaderboard Results: 0.807 is Seedance 2.0’s Overall score on the 115-sample joint leaderboard, exceeding Gemini-Omni-Flash by 0.063, while LTX-2.3 leads Keyframe Fidelity at 0.855 and Temporal Organization at 0.899 but ranks fourth overall at 0.659.Seedance 2.0’s advantage reflects balanced performance, whereas LTX-2.3’s stronger keyframe-control scores do not translate into the highest overall score.
  • Failure Modes: LTX-2.3 achieves the strongest open-source performance and highest keyframe response, but large scene, viewpoint, or shot-scale changes can cause abrupt transitions and rapid motion can produce intermediate-frame artifacts.These failures align with weaker minimal-mode DVQ (0.453) and PR (0.417) results relative to its keyframe response.
  • Failure Modes: Gemini-Omni-Flash illustrates that semantic compliance does not guarantee visual grounding: it ranks highly on VMA under segment-specific prompting while retaining relatively low KFS and often re-staging rather than continuing the input frames.The results show that instruction adherence, input preservation, temporal organization, and general video quality are noninterchangeable capabilities.
  • Constraint Density: VMA decreases by 0.060 to 0.260 under segment-specific prompts for all four proprietary models, and denser keyframe constraints further reduce instruction adherence while AMA remains stable.The degradation is concentrated in visual and temporal control as additional keyframes shorten intervals and impose more segment-level requirements.

5. Conclusion

KeyFrame-Compass introduces a comprehensive benchmark for keyframe-conditioned video generation, covering diverse generation scenarios through controlled variation in keyframe density, video structure, prompt settings, input formats, and application domains. Its evaluation framework combines keyframe response metrics that assess reproduction at intended temporal positions and durations.

  • KeyFrame-Compass introduces a comprehensive benchmark for keyframe-conditioned video generation.
  • 386 samples span four keyframe densities, two video structures, two prompt settings, two input formats, and three application domains.
  • The evaluation framework combines keyframe response metrics assessing whether each input keyframe is reproduced at its intended temporal position and duration.

KeyFrame-Compass: Towards Comprehensive Evaluation of

The section presents KeyFrame-Compass as a comprehensive evaluation framework for keyframe-conditioned video generation.

  • KeyFrame-Compass targets comprehensive evaluation of keyframe-conditioned video generation.

A. Case Study

The case study presents representative generated videos from open-source and proprietary models to illustrate their output quality and generation preferences. Examples are read in row-major order.

  • A. Case Study: Representative generated videos from open-source and proprietary models illustrate their output quality and generation preferences.All examples should be read in row-major order.

A.1. Performance of Proprietary Models

Proprietary models achieve strong qualitative performance but exhibit different generation preferences. Gemini-Omni-Flash may not faithfully reproduce input keyframes, while still producing coherent, fluent videos with complete story progression and good instruction alignment.

  • Overall performance: Proprietary models achieve strong qualitative performance but exhibit different generation preferences.The comparison covers minimal-prompt and segment-specific-prompt settings.
  • Gemini-Omni-Flash: Gemini-Omni-Flash does not always faithfully reproduce the input keyframes.
  • Gemini-Omni-Flash: Gemini-Omni-Flash produces coherent, fluent videos with complete story progression and good instruction alignment.

A.2. Performance of Open-Source Models

Among open-source models, LTX 2.3 performs best overall by recovering and reproducing most storyboard-grid keyframes, but its transitions remain rigid rather than continuous.

  • Open-Source Model Performance: LTX 2.3 achieves the strongest overall open-source performance, recovering and reproducing most input keyframes from the storyboard grid.Its examples span minimal-prompt and segment-specific-prompt settings.
  • Open-Source Model Performance: Its keyframe transitions often appear as hard cuts or slide-like changes instead of continuous video motion.This indicates a limitation in transition smoothness despite strong keyframe reproduction.

B. Calibration and Computation of Keyframe Similarity

The keyframe similarity metric calibrates a semantic matching threshold before test evaluation, combines semantic and pixel-level similarity for matched keyframes, and treats unmatched keyframes as zero contribution. Sensitivity analysis shows that KFS remains stable across tested pixel-score parameter ranges.

  • Semantic threshold calibration: The semantic threshold is calibrated on 1,027 inspectable validation pairs after excluding three uncertain cases from 1,030 candidate pairs.Validation samples span both prompt modes and four short-video proprietary models, using a relaxed criterion requiring a shared salient subject, object, or event participant.
  • Semantic threshold calibration: γ⋆= 0.80 is fixed before test evaluation, yielding a validation FPR of 5.3% and FNR of 6.4%.The equal-error operating point was γ = 0.7837, with FPR 5.3% and FNR 5.4%, before selecting the rounded conservative threshold.
  • KFS computation: KFS combines semantic and pixel-level similarities for each matched keyframe, while unmatched keyframes remain in the denominator and contribute zero.Pixel similarity uses PSNR and SSIM with lower anchors τp = 12.0 dB and τs = 0.65, plus saturation values of 50 dB and 1.0, respectively.
  • KFS computation: The pixel-score parameters do not determine semantic matching; they only define the scale of the continuous pixel score.Semantic matches are determined separately from the empirical PSNR and SSIM normalization anchors.
  • Sensitivity analysis: 0.0244 is the maximum mean KFS change across tested parameter settings, while 0.997 is the minimum Spearman rank correlation with the default setting.The sensitivity analysis varies τp ∈{10, 12, 14} dB, τs ∈{0.60, 0.65, 0.70}, and PSNR saturation in {40, 50, 60} dB.

C. MLLM Judge Sampling and Stability

The evaluation samples generated videos uniformly at 8 FPS for MLLM-judged metrics, preserving short-lived changes while keeping inputs manageable. Scores vary only slightly across 8, 4, and 2 FPS, and repeated five-run judgments assess stability on a 115-sample common set.

  • Frame sampling: 8 FPS sampling preserves short-lived motion changes and visual defects while keeping the judge input length manageable.All MLLM-judged metrics use frames sampled uniformly at 8 FPS and provided chronologically.
  • Frame sampling: Metric scores vary only slightly across 8, 4, and 2 FPS without changing the overall conclusions.The reported results therefore adopt 8 FPS.
  • Repeated-run stability: Five repeated MLLM evaluations at 8 FPS measure judgment stability on the 115-sample common set used by the six-model short-video leaderboard.The runs use identical video inputs, prompts, and judge configurations.
  • Repeated-run stability: Within each model and prompt mode, metric averages are computed over valid samples before aggregating Visual Quality and Spatiotemporal Coherence.Visual Quality is the mean of SVQ and DVQ, while Spatiotemporal Coherence is the mean of AC, SOC, and additional listed metrics.

D. Human Alignment Protocol

The human-alignment protocol compares five models across 60 shared cases using blinded three-annotator pairwise judgments, with a rubric aligned to the automatic evaluation dimensions. It aggregates preferences into model-level win rates and evaluates human–automatic agreement with decisive-majority filtering, clustered bootstrap intervals, and rank correlation.

  • Evaluation protocol: The evaluation subset contains 60 cases: 10 one-take and 50 multi-shot, spanning 3, 6, 9, and 12 keyframes.The keyframe-density counts are 20, 20, 10, and 10 cases, respectively.
  • Evaluation protocol: 600 pairwise comparison items are independently rated by three domain experts, who choose Video A, Video B, or Tie with model identities and automatic scores hidden.The comparisons cover ten unordered model pairs, using randomized left–right video order and the provided keyframes and segment-specific prompt.
  • Human rubric: The human rubric mirrors the automatic metrics’ observable scopes, combining visual-quality criteria into Video Quality and consistency and physical-reasoning criteria into Spatiotemporal Coherence.The rubric includes clarity, artifacts, stability, discontinuity, repetition, implausible morphing, flicker, attribute consistency, spatial orientation, and physical rationality.
  • Aggregation and alignment: Human win rates average preference contributions across four opponents, 60 cases, and three annotators, yielding 4 × 60 × 3 = 720 judgments per model.A selected model receives 1, its opponent 0, and ties contribute 0.5 to each; the overall human score averages the four dimension-level win rates.
  • Agreement analysis: Decisive human preferences require at least two of three annotators to select the same model, while agreement excludes non-decisive outcomes and uses 10,000 case-clustered bootstrap replicates for 95% confidence intervals.Decisive-item coverage is decisive comparisons divided by all 600 comparisons for each dimension; ties remain included in human win-rate aggregation.
  • Human–automatic alignment: Spearman’s ρs compares five model-level human win rates with automatic scores after ranking, with exact two-sided p-values obtained by enumerating all 5! = 120 label permutations.All annotations, including ties, contribute to human win rates, while the decisive-majority filter applies only to pairwise agreement.

E. Prompt Templates … E.3. Prompts for Checklist Generation

KeyFrame-Compass defines prompt templates for detailed and minimal keyframe-conditioned video generation, then aligns evaluation with the generated video's actual structure. Its checklist-generation prompts produce metric-specific inspection criteria without scoring the video, using structured references and strict response schemas.

  • E. Prompt Templates: Prompt templates cover segment-specific and minimal granularities for single-shot and multi-shot videos, with keyframes serving as ordered temporal anchors.Single-shot prompts add global camera motion and segment descriptions, while multi-shot prompts organize specifications at the shot level.
  • E.1. Prompt Templates for Video Generation: Minimal prompts specify keyframe order, story synopsis, video structure, and duration, leaving events and transitions mainly for the model to infer.The template also instructs the model to preserve keyframe appearance and generate natural motion, state changes, and transitions.
  • E.2. Prompts for Video Segmentation: Generated videos are segmented from observed cuts rather than requested structure, allowing keyframe requirements to match actual generated-video segments.Gemini 3.1 Pro performs segmentation, treating continuous takes as single shots and recognizing only visible edit boundaries.
  • E.2. Prompts for Video Segmentation: Keyframes are assigned only after shot boundaries are fixed, and assignments cannot add, remove, split, merge, shift, or reorder actual shots.Unmatched keyframes are placed in unassigned_keyframes rather than assigned without visible evidence.
  • E.2. Prompts for Video Segmentation: The segmentation output uses chronological shot and segment identifiers, contiguous non-overlapping time ranges covering the video, and explicit rules for repeated or unassigned keyframes.Actual single-shot videos use shot_id S1, while unassigned reasons are restricted to not_visible_in_generated_video or ambiguous.
  • E.3. Prompts for Checklist Generation: Checklist generation runs independently per metric, converting each sample's specification, keyframes, shot descriptions, hints, and metric procedure into human-readable inspection lenses.The generator defines atomic observable claims and reference keyframes but does not score the generated video or supply item-level pass/partial/fail rules.
  • E.3. Prompts for Checklist Generation: Checklist outputs follow schema v3 with dimension, reference_keyframe_ids, evidence_scope, human_question, and inspect fields, plus subject classification for Attribute Consistency.Metric procedures fix boundaries, valid dimensions, evidence scope, item granularity, and question style; specialized procedures cover static quality, dynamic quality, attributes, spatial orientation, and video adherence.

E.4. Prompts for Grouped Judge

The grouped judge evaluates related metrics together using generated videos, input keyframes, filtered context, and precomputed checklists. It grounds each score in timestamped observable evidence while preserving metric-specific results and normalized aggregation.

  • Grouped evaluation: The grouped judge resolves referenced keyframes through judge grounding and applies the matching rubric to every checklist item, while retaining separate results and scores per metric.The evaluation input includes the generated video, input keyframes, filtered evaluation context, and precomputed checklists.
  • Scoring procedure: Metric-level scores are normalized to [0, 1] and computed as the arithmetic mean of valid checklist-item scores, with insufficient evidence recorded as null.The response parser recomputes the metric-level score from valid check results.
  • Evidence contract: Each checklist item must be scored from observable evidence, with concrete observations specifying what occurred, where in time, and how it supports pass, partial, or fail.Generic judgments such as “looks good” are insufficient without timestamps and observable detail; time_range_sec is preferred over frame IDs when the full video is attached.
  • Metric rubrics: The rubric section defines ownership boundaries and normalized score anchors for each metric without adding checklist fields.The grouped template fills the rubric_section placeholder with the applicable metric rubrics.
Loading 2607.14202v1…