PaperlayerView exact revision on arXiv

Source-linked AI summary

StreamAV-Bench: A Comprehensive Benchmark for Streaming Audio-Video Generation

Kaiqi Liu, Haoxuan Zeng, Jingqi Liu, Jiacong Fang, Ziqi Cai, Yunyao Mao, Henglin Liu, Yu Sheng, Shuchen Weng, Boxin Shi

arXiv:2608.26336v1cs.SDcs.CVcs.MM

TL;DR

Existing benchmarks largely assess completed audio-video outputs, leaving streaming properties undercovered. StreamAV-Bench evaluates progressive and interactive generation with expert-verified cases across 32 dimensions. Its evaluation finds complementary strengths across paradigms, with no single system dominating the full suite.

  • Problem

    Existing benchmarks largely assess completed audio-video outputs and provide limited coverage of streaming properties.

    Source: §1 · p. 2

  • Method

    StreamAV-Bench evaluates progressive and interactive generation through expert-verified cases and a unified framework spanning 32 fine-grained dimensions.

    Source: §1 · p. 2

  • Results

    Native joint models and cascaded pipelines show complementary strengths across evaluation dimensions, and no single system dominates the full suite.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    The findings point to long-horizon stability and real-time interactive capabilities as development targets for native joint audio-video streaming models.

    Source: §1 · p. 2

Abstract

from arXiv · show

Recent advancements in generative models are pushing video generation toward unbounded streaming audio-video generation for real-time interactive worlds. However, existing benchmarks primarily evaluate completed sequences and struggle to capture streaming properties. To bridge this gap, we introduce StreamAV-Bench, the first comprehensive benchmark tailored for streaming audio-video generation. StreamAV-Bench establishes a unified evaluation framework, including the progressive track for instruction adherence and long-horizon stability, and the interactive track for interactive response and state retention and reuse. With expert-verified evaluation cases across 32 fine-grained dimensions, we conduct an extensive evaluation of 13 representative systems. Our analysis reveals that current models suffer from temporal drift in progressive generation and responsiveness bottlenecks during interactive control. Based on a comprehensive failure analysis, we share insights to advance the development of native joint audio-video streaming models.

1 INTRODUCTION

StreamAV-Bench evaluates streaming audio-video generation through progressive and interactive tracks, a unified 32-dimension framework, and an assessment of 13 representative systems. Its evaluation highlights long-horizon degradation and interactive-response bottlenecks.

1 INTRODUCTION

  • Streaming generation requires maintaining audio-video quality and cross-modal alignment, avoiding temporal drift, responding rapidly to dynamic world-state changes, and preserving boundary continuity.Generation proceeds over an expanding time horizon while users provide free-form prompts and previously generated clips remain fixed as committed history.Source: §1 · p. 2
  • Existing benchmarks miss streaming properties: video-centric benchmarks lack audio-video quality assessment, while audio-video benchmarks mainly assess completed outputs and basic metrics.Prior work explores long-horizon generation (Huang et al. (2025); Zheng et al. (2025); Yuan et al. (2026); Liu et al. (2026a); Zhang et al. (2026)), audio-video quality and alignment (Hua et al. (2025); Cao et al. (2025); Zhou et al. (2026b)), with sparse work on long horizons (Liu et al. (2026c)) and scripted narratives (Wei et al. (2026)).Source: §1 · p. 2
  • StreamAV-Bench introduces progressive and interactive tracks across 320 expert-verified scenarios, each generated continuously for 180 seconds.The progressive track tests instruction adherence and long-horizon stability; the interactive track tests response, state retention, and reuse. Scenarios span 8 scene domains, 5 audio domains, 5 subject categories, and 4 visual styles.Source: §1 · p. 2
  • The unified framework assesses 32 fine-grained dimensions covering track-specific capabilities, streaming continuity and efficiency, and overall audio-video quality and alignment.Content complexity covers entities, activities, and audio; interactive update complexity covers interaction type, update modality, and temporal dependency.Source: §1 · p. 2
  • Evaluation of 13 representative systems reveals complementary strengths in cascaded and native joint systems, dimension-specific long-horizon degradation, and a mismatch between update achievement and response latency.The analysis also identifies modality-driven interaction bottlenecks and includes qualitative failure analysis.Source: §1 · p. 2

2 RELATED WORK

Prior work has advanced streaming audio-video generation and evaluated video or audio-video outputs across diverse qualities and settings, but existing benchmarks do not directly address streaming systems for real-time interactive worlds.

2.1 STREAMING AUDIO-VIDEO GENERATION

  • Streaming audio-video work combines cascaded audio addition or joint generation with progressive video techniques, synchronization, and controlled multi-shot or long-horizon generation.Causal distillation and self-rollout reduce accumulated errors; block-wise processing, rolling contexts, and bounded caches are also used. Related works include Cheng et al. (2024); Shan et al. (2025); Ruan et al. (2022); Liu et al. (2025a); Low et al. (2025); HaCohen et al. (2026); Yin et al. (2024); Huang et al. (2026b); Kodaira et al. (2025); Yuan et al. (2026); Liu et al. (2025b); Yang et al. (2025); Su et al. (2026); Huang et al. (2026a); Liu et al. (2026b); and Shi et al. (2026).Source: §2.1 · p. 3

2.2 VIDEO GENERATION BENCHMARKS

  • Video benchmarks assess quality, consistency, prompt alignment, compositional following, dynamics, state transitions, long-video degradation, narrative alignment, and long-range memory, but remain video-centric.Examples include FETV and EvalCrafter for human-aligned evaluations, VBench++ across settings, and benchmarks targeting long-video or multi-prompt evaluation. Cited works: Huang et al. (2023); Liu et al. (2023b); Liu et al. (2023a); Huang et al. (2025); Sun et al. (2024); Liao et al. (2024); Feng et al. (2025a); Zhang et al. (2025); Zheng et al. (2025); Liu et al. (2026a); and Zhang et al. (2026).Source: §2.2 · p. 3

2.3 AUDIO-VIDEO GENERATION BENCHMARKS

  • Audio-video benchmarks cover output quality, cross-modal semantics, synchronization, instruction following, realism, and specialized settings, yet evaluate static completed generations rather than streaming interaction.MTAVG-Bench and FoleyBench target visually grounded sound and multi-talker scenarios; MSAVBench covers multi-shot generation, and LongAV-Compass examines minute-long generation under diverse conditioning. Cited works: Mao et al. (2024); Hua et al. (2025); Cao et al. (2025); Zhou et al. (2026b); Zhou et al. (2026a); Dixit et al. (2025); Wei et al. (2026); Liu et al. (2026c).Source: §2.3 · pp. 3–4

3 STREAMAV-BENCH

StreamAV-Bench defines a benchmark for progressive and interactive streaming audio-video generation, with structured scenarios and a 32-dimension evaluation framework. It tests long-horizon generation, prompt-driven updates, and whether established state is retained and reused.

3.1 TASK DEFINITION

  • The progressive track tests instruction adherence and stability over 180 seconds, while the interactive track tests responses and state retention and reuse across five updates.Both tracks are also evaluated at 60-, 120-, and 180-second horizons. Interactive prompts arrive every 30 seconds and may update visual content, audio content, or both.Source: §3.1 · p. 4

3.2 BENCHMARK CONSTRUCTION

  • Benchmark construction combines expert-designed taxonomy and scenario allocation with scripts, interval annotations, prompts, checklists, and expert review.The 160 themes are each used in both tracks, yielding 320 scenarios. An LLM generates prompts and verifiable checklists from annotated scripts (OpenAI (2026)); two independent experts review cases and a third adjudicates disagreements, producing the final curated set.Source: §3.2 · pp. 4–5

3.3 BENCHMARK STATISTICS

  • The 320 scenarios span 8 scene domains, 5 audio domains, 5 subject categories, and 4 visual styles, with complexity dimensions tailored to content and interactive updates.The benchmark includes 192 photorealistic and 128 non-photorealistic scenarios. Content complexity covers entity, activity, and audio; interactive update complexity covers interaction type, update modality, and temporal dependency.Source: §3.3 · pp. 5–6

3.4 EVALUATION FRAMEWORK

  • The evaluation framework combines expert models and checklist-guided MLLM assessments across 32 dimensions covering six categories.Progressive measures instruction fulfillment and drift, plus long-horizon stability; interactive measures update fulfillment, continuity, response latency, state retention, and history-dependent reuse. The framework also assesses visual, audio, and cross-modal quality and alignment. State retention accounts for requested changes, while history-dependency following checks whether earlier content is used correctly.Source: §3.4 · pp. 6–7
  • Shared evaluation measures streaming continuity and efficiency alongside quality and cross-modal alignment.Native boundary continuity detects artifacts between generation units, while frames per second and time-to-first-chunk capture generation speed and startup latency. Quality and alignment assessments cover visual and audio quality, semantic correspondence, and temporal synchronization.Source: §3.4 · pp. 6–7

4 EXPERIMENTS

Experiments evaluate 13 streaming audio-video systems across progressive and interactive tracks. Results reveal paradigm-specific strengths, temporal and interaction bottlenecks, and strong alignment between benchmark scores and human preferences.

4.1 EXPERIMENTAL SETUP

  • The evaluation covers 13 systems across native joint and cascaded streaming paradigms, using case-specific checklists and track-specific metrics with uniform averaging across cases.Systems process identical inputs: a global prompt for progressive generation and ordered prompts at fixed 30-second intervals for interaction; open-source systems use official checkpoints and pipelines, and commercial systems their official web interfaces.Source: §4.1 · p. 7

4.2 MAIN RESULTS

  • Native joint systems lead in audio fidelity, synchronization, and most interactive metrics, while cascaded pipelines lead in visual quality, alignment, long-horizon stability, and continuity; no system dominates.Instruction adherence is mixed: native systems lead in video instruction fulfillment and audio instruction drift, while cascaded pipelines lead in audio instruction fulfillment and video instruction drift.Source: §4.2 · p. 7, Table 2 · p. 8
  • IAMFlow’s highest VQ (2.840) coincides with VQ-D of 0.425, while Causal-Forcing’s lowest VQ-D (0.189) accompanies VQ of 2.397; fast responses also do not ensure successful updates.HappyOyster has the highest PVUAR and PAUAR (0.518 and 0.488) but latencies of 14.484 and 13.621 seconds; SWIFT and OmniForcing have the lowest PVRL (10.340 seconds) and PARL (7.925 seconds), but lower corresponding achievement rates. MemFlow has the highest VSR (2.961), and no system exceeds 2.817 in HDF.Source: §4.2 · pp. 7–8

4.3 ANALYSIS AND FINDINGS

  • Over 180 seconds, visual aesthetics, AVAlign, and VQ degrade in 13, 12, and 10 systems, while PQ, AQ, and AVSync show mixed directions.Odyssey-2 and PixVerse R1 have improving VQ and AQ trends (−0.056 and −0.019) but the largest respective variations (0.423 and 0.508); SWIFT has the steepest AVSync degradation (0.062) and largest variation (0.536).Source: §4.3 · p. 9, Figure 4 · p. 9
  • Longer rollouts increase progressive quality drift, while interactive degradation is mainly audio-side, with slower responses and declining audio state retention.From 60 to 180 seconds, VA-D rises from 0.025 to 0.042 and VQ-D from 0.313 to 0.361; AUF falls from 2.481 to 2.332, PAUAR from 0.361 to 0.331, and ASR from 3.022 to 2.895, while both response latencies increase by about 1.3 seconds.Source: §4.3 · p. 9
  • Increasing content complexity primarily impairs visual realization: dense entities reduce VIF from 3.090 to 2.816, and concurrent activities lower VQ by 0.129, while consistency remains nearly unchanged.Multi-source cases have the lowest AIF (2.523); entity density also lowers VQ from 2.760 to 2.651, whereas subject and background consistency remain nearly unchanged across factors.Source: §4.3 · p. 9
  • Interactive bottlenecks include visual updates, state changes, and history reuse; qualitative failures also include drift, degradation, inconsistency, and discontinuities at prompt-update boundaries.Visual updates have lower achievement and slower response than audio updates (PVUAR/PAUAR: 0.243/0.331; PVRL/PARL: 13.534/12.437 seconds); HDF is 2.477 for adjacent and 2.321 for long-range dependencies. Figure 5 groups eight common failure modes across progressive generation and interaction.Source: §4.3 · pp. 9–10, Figure 5 · p. 10

4.4 HUMAN ALIGNMENT

  • Benchmark scores align strongly with human preferences: all 17 evaluated metrics show model-level Spearman correlations from 0.77 to 0.94.Blind pairwise comparisons used a balanced 60-case subset spanning both tracks and all 13 systems, with three annotators comparing anonymized outputs and ties allowed.Source: §4.4 · p. 10, Table 3 · p. 10

5 INSIGHTS AND FUTURE DIRECTIONS

The paper identifies stateful streaming as the central paradigm for future audio-video generation, shifting from discrete clips toward continuously evolving worlds that can be maintained, updated, and interacted with in real time.

Stateful streaming

  • Future audio-video models should maintain persistent world state, update it dynamically, and render sequences with minimal latency.This shifts the field from discrete clip generation toward continuously evolving audio-video world models.Source: §5 · pp. 10–11

Maintaining long-horizon stability

  • Long-horizon generation should preserve perceptual quality, instruction adherence, subject appearance, and audio-video alignment throughout an unbounded stream.Training and inference should jointly optimize absolute performance and long-horizon stability rather than clip-level objectives.Source: §5 · p. 10

Advancing reliable interaction

  • Reliable interaction requires runtime updates to modify specified content while retaining valid elements and reusing relevant information from earlier interactions.Selective state-update mechanisms should jointly optimize update achievement, state retention and reuse, and response latency for robust free-form interaction.Source: §5 · p. 10

Developing joint audio-video streaming

  • Future systems should generate audio and video jointly and incrementally from a shared, continuously updated latent state.This native architecture should be paired with state caching and coordinated audio-video scheduling to minimize latency while maintaining synchronization.Source: §5 · p. 11

6 CONCLUSION · A BENCHMARK DETAILS · A.1 TASK SETTINGS

StreamAV-Bench provides a unified benchmark for long-horizon and interactive streaming audio-video generation, evaluating 13 systems across expert-verified scenarios and 32 dimensions. Its progressive and interactive tracks test single-prompt stability and response to runtime updates over 180-second rollouts.

6 CONCLUSION

  • StreamAV-Bench covers 320 expert-verified scenarios across progressive and interactive tracks, evaluating 13 native joint audio-video systems and cascaded pipelines under one protocol.The benchmark supports systematic comparison and reveals failure modes that existing benchmarks cannot capture.Source: §6 · p. 11
  • Its 32-dimensional framework measures quality and alignment, continuity and efficiency, instruction adherence, long-horizon stability, interactive response, and state retention and reuse.Source: §6 · p. 11

A.1 TASK SETTINGS

  • Both tracks contain 160 scenarios and generate 180-second audio-video outputs from a single rollout.Source: §A.1 · p. 15
  • The progressive track tests instruction adherence and long-horizon stability using a 180-second sequence generated from one prompt describing a global state.Source: §A.1 · p. 15
  • The interactive track tests response and state retention and reuse with an ordered prompt sequence: an initial scenario prompt followed by five updates every 30 seconds.Updates can change visual content, audio content, or both.Source: §A.1 · p. 15
  • For both tracks, evaluation covers the first 60, 120, and 180 seconds of the same rollout.Source: §A.1 · p. 15

A.2 BENCHMARK CONSTRUCTION · A.3 BENCHMARK STATISTICS

StreamAV-Bench builds expert-verified progressive and interactive scenarios from a shared taxonomy, with scripts, prompts, and evaluation materials designed to preserve temporal and state consistency. Its statistics characterize semantic coverage, content complexity in both tracks, and interaction complexity across the interactive track’s runtime updates.

A.2 BENCHMARK CONSTRUCTION

  • 160 scenario themes appear once in each track, sharing semantic scope but independently realized and allocated across domains, subjects, styles, complexity levels, and interactive-update attributes.Interactive runtime updates are distributed by interaction type, update modality, and temporal dependency.Source: §A.2 · p. 15
  • Progressive scripts define open-ended audio-video processes for 180-second rollouts, while interactive scripts jointly plan the initial world and all five runtime updates as one state trajectory.The interactive updates are planned together rather than constructed independently.Source: §A.2 · p. 15
  • Each interactive update has one primary intent and annotations for interaction type, modality, and temporal dependency, within a causally ordered, state-consistent sequence.Adjacent updates depend on the immediately preceding update; long-range updates depend on a non-adjacent earlier one, and existing entities, activities, sounds, and states persist unless explicitly changed.Source: §A.2 · p. 15
  • GPT-5.4 OpenAI (2026) generates prompts from the scripts and annotations: one global prompt per progressive scenario, or an initial prompt plus five concise runtime updates per interactive scenario.All prompts use a common, generation-oriented style.Source: §A.2 · p. 15
  • GPT-5.4 derives modality-specific visual and audio checklists, with interactive updates serving as evaluation targets and dependent updates linked to their earlier prompts.Joint audio-video updates are split into visual and audio targets while preserving their original semantics.Source: §A.2 · pp. 15–16
  • Two domain experts independently review scripts, prompts, checklists, targets, and dependencies; a third adjudicates disagreements, yielding 320 expert-verified scenarios before model evaluation.Review checks taxonomy and complexity alignment, update order and dependencies, feasibility, audio-video plausibility, coverage, state consistency, and evaluation consistency; cases needing correction are revised and reviewed again.Source: §A.2 · p. 16

A.3 BENCHMARK STATISTICS

  • The 320 scenarios span 8 scene domains, 5 audio domains, 5 subject categories, and 4 visual styles, including 192 photorealistic and 128 non-photorealistic scenarios.Subject categories reflect each scenario’s primary subject, while audio domains reflect its principal audio requirement.Source: §A.3 · p. 16
  • Both tracks are balanced across three entity, activity, and audio complexity levels, while interactive updates are characterized by interaction type, modality, and temporal dependency.Per track: entity complexity has 56 Single, 72 Multi, and 32 Dense scenarios; activity complexity has 40 Atomic, 80 Relational, and 40 Concurrent; audio complexity has 48 Single-Source, 64 Multi-Source, and 48 Overlapping. Across 800 updates: there are 440 agent actions, 160 entity-state changes, 80 environment changes, and 120 audio changes; 360 video-only, 160 audio-only, and 280 joint audio-video updates; and 440 independent, 240 adjacent, and 120 long-range dependencies.Source: §A.3 · p. 16

B EVALUATION FRAMEWORK DETAILS … B.6 STATE RETENTION AND REUSE

The framework evaluates progressive generation and interactive control across 32 metrics, combining expert models with checklist-guided MLLM assessments. Shared metrics cover quality, alignment, continuity, and efficiency; observations are generally averaged uniformly across cases.

B EVALUATION FRAMEWORK DETAILS

  • The 32 metrics span progressive instruction adherence and long-horizon stability, interactive response and state retention, and shared streaming and content-quality properties.The framework combines tailored expert models with checklist-guided MLLM assessments; unless otherwise stated, observations are aggregated within cases and averaged uniformly across cases.Source: B EVALUATION FRAMEWORK DETAILS · pp. 16–17

B.1 QUALITY AND ALIGNMENT

  • Visual Aesthetics, Visual Quality, and Production Quality assess visual appeal, fidelity, and production quality, using normalized or evaluator-native scores where specified.The six quality and alignment metrics are computed over six non-overlapping 30-second intervals and averaged within each case; VA uses a 0–1 scale, while VQ averages four dimensions on a 1–5 scale.Source: §B.1 · p. 17
  • Audio Quality assesses naturalness and artifacts, while Audio-Video Alignment measures semantic correspondence and Audio-Video Synchronization measures temporal offset.AQ averages two dimensions on a 1–5 scale; AVAlign uses cosine similarity in ImageBind space, while AVSync reports Synchformer-predicted offsets in seconds, with higher better for AVAlign and lower better for AVSync.Source: §B.1 · p. 17

B.2 STREAMING CONTINUITY AND EFFICIENCY

  • Native Boundary Continuity penalizes black frames, repeated frames, freezes, and flashes at model-native boundaries, while FPS and Time-to-First-Chunk measure generation speed and initial-output delay.NBC combines the four artifact measures in a clipped 0–100 score averaged over boundaries and cases; higher is better, as is FPS, while lower TTFC is better.Source: §B.2 · p. 17

B.3 INSTRUCTION ADHERENCE

  • Fixed visual and audio checklists define progressive-case requirements; fulfillment scores assess realization, while drift scores compare adherence between the first and last intervals.VIF covers scene, style, subjects, and activity; AIF covers sound source, content, and temporal relations. Gemini scores applicable criteria from 1–5, while VID/AID average matched-criterion endpoint changes on a 0–4 scale, with lower drift better.Source: §B.3 · pp. 17–18

B.4 LONG-HORIZON STABILITY

  • Long-horizon stability measures endpoint quality drift, subject and background consistency, and audio-video semantic and synchronization drift across the generation horizon.Drift compares the first and last 30-second intervals: quality and audio-video drift are lower-is-better absolute changes, while subject and background consistency are higher-is-better scores.Source: §B.4 · p. 18

B.5 INTERACTIVE RESPONSE

  • Interactive response evaluates runtime updates at five boundaries, scoring target fulfillment and visual or audio continuity while allowing requested changes.VUF/AUF score realization of update targets; PVC combines boundary-artifact scoring with MLLM judgments, and PAC evaluates audio dropouts, cuts, repetition, clicks, clipping, and abrupt volume changes.Source: §B.5 · pp. 18–19
  • Update achievement rates count clearly realized applicable targets, while response latency measures time to first clear realization conditional on achievement.PVUAR/PAUAR rates range from 0 to 1; PVRL/PARL are reported in seconds on the generated-content timeline and averaged over successful updates and cases with at least one achievement.Source: §B.5 · p. 19

C IMPLEMENTATION DETAILS … D.5 TEMPORAL EVOLUTION

The paper standardizes evaluation across 13 streaming audio-video systems and finds that performance varies by interactive metric, while longer streams expose progressive quality drift and interactive audio-side degradation.

C.1 EVALUATED SYSTEMS

  • The evaluation covers 13 systems: native joint streaming audio-video models and cascaded pipelines pairing streaming video generators with HunyuanVideo-Foley Shan et al. (2025).Open-source systems use official checkpoints and default recommended configurations on NVIDIA H800 GPUs, while commercial systems use official web interfaces; NBC, FPS, and TTFC are unavailable for PixVerse R1, HappyOyster, and Odyssey-2.Source: §C.1 · p. 19, Table A2 · p. 20

C.2 INTERACTIVE CONTINUATION PROTOCOL / C.3 CASCADED AUDIO GENERATION / C.4 GENERATION CONTROLS

  • All systems receive identical inputs: one global prompt for progressive generation and ordered prompts at fixed 30-second intervals for interactive generation, with outputs converted to a common audio-video format.For the shared runtime-update protocol, previously generated frames remain unchanged as conditioning switches to each active prompt; cascaded audio segments follow the corresponding video prompt schedule.Source: §C.4 · p. 20, §C.2 · p. 20, §C.3 · p. 20

D.1 INITIAL-WORLD ALIGNMENT

  • HappyOyster has the highest Initial VIF, while SWIFT has the highest Initial AIF, so no single system leads both modalities.Scores apply VIF/AIF checklists derived from the initial prompt over the first interval and are averaged across applicable criteria and 160 interactive cases.Source: §D.1 · pp. 20–21, Table A3 · p. 21

D.2 UPDATE ACHIEVEMENT AND RESPONSE LATENCY

  • HappyOyster scores 0.518 PVUAR and 0.488 PAUAR, with PVRL of 14.484 seconds and PARL of 13.621 seconds.SWIFT’s lower PVRL (10.340 seconds) accompanies PVUAR of 0.246, while OmniForcing’s lowest PARL (7.925 seconds) accompanies PAUAR of 0.316; latency is measured over achieved targets, so lower latency alone does not mean stronger overall responsiveness.Source: §D.2 · p. 21, Table A4 · p. 21

D.3 HISTORY-DEPENDENCY SOURCE ESTABLISHMENT

  • HappyOyster leads adjacent source establishment and both conditional HDF scores, while SWIFT leads long-range source establishment.OmniForcing ranks second in conditional HDF-A despite the lowest adjacent establishment rate, showing that source establishment and dependency following capture distinct failure modes.Source: §D.3 · pp. 21–22, Table A5 · p. 22

D.4 SUCCESS- AND COVERAGE-ADJUSTED DIAGNOSTICS

  • With penalties for unachieved updates and unestablished dependency sources, HappyOyster has the lowest SA-PVRL and highest CA-HDF-A and CA-HDF-L, while PixVerse R1 has the lowest SA-PARL.Unachieved updates receive the 30-second observation-window cap; adjusted latencies remain media-time latencies, and coverage-adjusted HDF scores combine source establishment with normalized conditional quality.Source: §D.4 · pp. 22–23, Table A6 · p. 23

D.5 TEMPORAL EVOLUTION

  • Visual aesthetics, audio-video alignment, and visual quality degrade in 13, 12, and 10 of 13 systems, while SWIFT has the steepest AVSync degradation (0.062).Odyssey-2 and PixVerse R1 show improving VQ and AQ trends (−0.056 and −0.019) but the largest variation in those dimensions (0.423 and 0.508); SWIFT also has the largest AVSync variation (0.536).Source: §D.5 · p. 23
  • Across 13 systems, progressive VA-D rises from 0.025 to 0.036 and 0.042, while VQ-D rises from 0.313 to 0.347 and 0.361 at 60, 120, and 180 seconds.Interactive AUF falls from 2.481 to 2.340 and 2.332, PAUAR from 0.361 to 0.333 and 0.331, and ASR from 3.022 to 2.952 and 2.895; VUF, PVUAR, and VSR remain stable, while PVRL and PARL each increase by about 1.3 seconds.Source: §D.5 · p. 24, Table A7 · p. 24

D.6 CONTENT AND UPDATE COMPLEXITY RESULTS

Progressive content complexity primarily degrades visual instruction fulfillment and quality, while subject and background consistency remain stable. Interactive generation is bottlenecked by visual updates, state revision, and reuse of earlier information.

Progressive track

  • Dense entities reduce visual instruction fulfillment, and increasing activity complexity lowers visual quality.VIF decreases from 3.090 for single-entity cases to 2.816 for dense cases; VQ decreases monotonically from 2.760 to 2.651 and by 0.129 from atomic to concurrent activities. Subject consistency and background consistency remain nearly unchanged across the three factors.Source: §D.6 · p. 24
  • Multi-source cases have the lowest audio instruction fulfillment, while concurrent activities do not reduce it correspondingly.AIF is 2.523 for multi-source cases; activity complexity lowers VQ without a corresponding decline in AIF.Source: §D.6 · p. 24

Interactive track

  • Visual updates have lower achievement rates and slower responses than audio updates despite comparable fulfillment.VUF and AUF are 2.347 vs. 2.332; PVUAR/PAUAR are 0.243/0.331, and PVRL/PARL are 13.534/12.437 seconds.Source: §D.6 · p. 24
  • Direct agent actions achieve higher visual update fulfillment than entity-state or environment changes, while long-range history reuse is weaker.VUF is 2.418 for direct agent actions versus 2.274 for entity-state changes and 2.283 for environment changes. HDF is 2.477 for adjacent and 2.321 for long-range dependencies.Source: §D.6 · p. 24

E HUMAN ALIGNMENT · F CHECKLIST CONSTRUCTION TEMPLATES

Human judgments align strongly with benchmark scores across all 17 metrics, while the appendix specifies structured templates for visual and audio checklists and for splitting joint runtime updates. The templates constrain questions and updates to prompt-stated requirements and preserve their semantics.

E HUMAN ALIGNMENT

  • The blind comparison protocol sampled 120 balanced pairs per metric, yielding 2,040 tasks and 6,120 judgments across 17 metrics.It covered 60 cases across both tracks and 13 systems; three annotators judged each pair independently, ties counted as half a win for each system, and majority vote set labels.Source: E HUMAN ALIGNMENT · pp. 24–25
  • Human alignment correlations span 0.77–0.94 across all 17 metrics; PVC is highest, while PVRL and PARL correlate less strongly than achievement and fulfillment metrics.Table A8 reports Spearman correlations between per-model human win rates and direction-adjusted benchmark scores.Source: Table A8 · p. 25, E HUMAN ALIGNMENT · p. 25

F CHECKLIST CONSTRUCTION TEMPLATES

  • The fixed LLM templates construct visual and audio checklists and decompose joint audio-video runtime updates, with curly-brace placeholders for case-specific content.The templates are provided for the finalized checklists and update decomposition.Source: F CHECKLIST CONSTRUCTION TEMPLATES · p. 25

F.1 CHECKLIST CONSTRUCTION AND UPDATE DECOMPOSITION

  • The visual instruction checklist organizes Video Instruction Fulfillment into scene, visual style, subjects, and activity fields.It generates concise, self-contained, objectively verifiable Yes/No questions only for explicitly stated visual requirements; unsupported requirements are excluded and inapplicable fields are null.Source: §F.1 · p. 25
  • The audio instruction checklist organizes Audio Instruction Fulfillment into sound source, sound content, and temporal relation fields.Questions address only explicitly stated auditory requirements, distinguish who or what produces a sound from what is heard, and use null when a field has no applicable requirement.Source: §F.1 · p. 26
  • The joint audio-video update splitter separates explicitly requested visual and audio changes while preserving original wording and semantic details where possible.It avoids rewriting beyond what is needed for the split or inferring new content.Source: §F.1 · pp. 26–27

G EXAMPLE BENCHMARK CASES · G.1 PROGRESSIVE EXAMPLE · G.2 INTERACTIVE EXAMPLE

The examples show how the benchmark uses track-specific prompt structures and temporal dependencies: a progressive sailing scenario and an interactive bread-baking schedule.

G EXAMPLE BENCHMARK CASES

G.1 PROGRESSIVE EXAMPLE

  • The progressive case depicts a sailor crossing an open bay with a border collie aboard.Source: §G.1 · p. 27
  • Its global prompt combines continuous steering and sail trimming, wind-driven boat motion, and a layered maritime soundscape.Source: §G.1 · p. 27
  • Table A9 provides the complete 180-second prompt example for the progressive track.Source: Table A9 · p. 27, §G.1 · p. 27

G.2 INTERACTIVE EXAMPLE

  • The interactive case follows bread baking in a photorealistic home kitchen, with Table A10 providing the complete 180-second prompt schedule.Source: §G.2 · p. 27, Table A10 · p. 27
  • The schedule links the 30-second dial adjustment to higher heat at 60 seconds and the crust color at 150 seconds, while closing the window at 120 seconds responds to rain introduced at 90 seconds.Source: §G.2 · p. 27

Related papers

Loading 2608.26336v1…