PaperlayerView exact revision on arXiv

Source-linked AI summary

StreamAV-Bench: A Comprehensive Benchmark for Streaming Audio-Video Generation

Kaiqi Liu, Haoxuan Zeng, Jingqi Liu, Jiacong Fang, Ziqi Cai, Yunyao Mao, Henglin Liu, Yu Sheng, Shuchen Weng, Boxin Shi

arXiv:2608.26336v1cs.SDcs.CVcs.MM

TL;DR

Existing benchmarks inadequately capture the streaming properties required by unbounded interactive audio-video generation. StreamAV-Bench addresses this gap with expert-verified progressive and interactive tracks, a 32-dimensional framework, and evaluation of 13 systems. The study reports complementary system strengths, long-horizon degradation, and responsiveness bottlenecks.

  • Problem

    Existing benchmarks primarily evaluate completed sequences and do not comprehensively assess streaming properties such as long-horizon stability, interactive response, and state retention.

  • Method

    StreamAV-Bench uses progressive and interactive tracks, expert-verified scenarios, and a unified framework with 32 fine-grained evaluation dimensions.

  • Results

    Evaluation of 13 representative systems reveals complementary strengths between native joint and cascaded systems, dimension-specific long-horizon degradation, and a mismatch between update achievement and response latency.

  • Takeaways & Limitations

    The benchmark supports systematic comparison of streaming audio-video systems and provides insights for developing native joint models with long-horizon stability and real-time interactive capabilities.

Abstract

from arXiv · show

Recent advancements in generative models are pushing video generation toward unbounded streaming audio-video generation for real-time interactive worlds. However, existing benchmarks primarily evaluate completed sequences and struggle to capture streaming properties. To bridge this gap, we introduce StreamAV-Bench, the first comprehensive benchmark tailored for streaming audio-video generation. StreamAV-Bench establishes a unified evaluation framework, including the progressive track for instruction adherence and long-horizon stability, and the interactive track for interactive response and state retention and reuse. With expert-verified evaluation cases across 32 fine-grained dimensions, we conduct an extensive evaluation of 13 representative systems. Our analysis reveals that current models suffer from temporal drift in progressive generation and responsiveness bottlenecks during interactive control. Based on a comprehensive failure analysis, we share insights to advance the development of native joint audio-video streaming models.

1 INTRODUCTION

StreamAV-Bench addresses the lack of benchmarks for streaming audio-video properties by introducing progressive and interactive tracks within a unified 32-dimensional framework. It evaluates 13 systems and finds complementary strengths, long-horizon degradation, and interactive responsiveness bottlenecks.

  • Existing benchmarks mainly evaluate completed sequences, omitting streaming requirements such as temporal stability, dynamic state updates, and interactive response.
  • StreamAV-Bench introduces progressive and interactive tracks for instruction adherence, long-horizon stability, interactive response, and state retention and reuse.
  • The benchmark provides a unified framework with 32 fine-grained dimensions covering progressive, interactive, continuity, efficiency, quality, and alignment metrics.
  • The study evaluates 13 representative systems, including native streaming audio-video models and cascaded T2V and V2A pipelines.
  • Native and cascaded systems show complementary strengths, while evaluations reveal dimension-specific long-horizon degradation and a mismatch between update achievement and response latency.

2 RELATED WORK

Prior video and audio-video benchmarks broaden coverage of quality, alignment, consistency, and generation duration, but remain designed primarily for static or completed outputs. StreamAV-Bench targets the omitted streaming setting.

  • Video benchmarks evaluate generated quality, temporal consistency, and prompt alignment, but generally lack audio-video quality assessment.
  • Audio-video benchmarks cover quality, cross-modal semantics, synchronization, instruction following, and specialized settings such as visually grounded sound and multi-talker scenarios.
  • StreamAV-Bench is introduced to evaluate real-time streaming systems whose outputs evolve progressively and respond to interactive prompts.
  • Existing benchmarks extend to multi-shot and minute-long outputs, yet remain fundamentally designed for static and completed generations.

3 STREAMAV-BENCH

StreamAV-Bench constructs expert-verified progressive and interactive scenarios and evaluates streaming generation across task-specific and shared dimensions. Its framework combines structured prompts, checklists, expert review, and multimodal assessment.

  • Task definition: Streaming generation incrementally establishes audio-video content as committed history, with prompts either fixed for progressive streaming or updated interactively.
  • Task definition: The progressive track uses one global prompt for 180 seconds, while the interactive track uses an initial prompt plus five 30-second updates and evaluates 60-, 120-, and 180-second rollouts.
  • Benchmark construction: The construction pipeline defines balanced taxonomies, scripts temporal annotations, generates prompts and verifiable checklists, and applies independent expert review with adjudication.
  • Benchmark statistics: The benchmark spans 8 scene domains, 5 audio domains, 5 subject categories, and 4 visual styles across 320 scenarios.
  • Evaluation framework: Its 32 evaluation dimensions cover instruction adherence, long-horizon stability, interactive response, state retention and reuse, continuity, efficiency, quality, and alignment.
  • Evaluation framework: Quality and alignment assessment separates visual quality, audio quality, and cross-modal alignment, while progressive and interactive metrics target their respective streaming behaviors.

4 EXPERIMENTS

Experiments evaluate 13 streaming systems under a unified protocol and show complementary strengths across native joint and cascaded paradigms, alongside temporal drift and interaction bottlenecks.

  • Main results: Native joint models lead audio fidelity, synchronization, and nine of eleven interactive metrics, while cascaded pipelines lead visual quality, semantic alignment, and most stability metrics.No single system dominates the full evaluation suite.
  • Main results: 2.840 VQ is paired with 0.425 VQ-D for IAMFlow, whereas Causal-Forcing reaches 0.189 VQ-D with 2.397 VQ.Absolute visual quality and temporal drift can favor different systems.
  • Analysis and findings: 0.062 AVSync degradation and 0.536 variation make SWIFT the most unstable system on synchronization.Visual aesthetics, alignment, and visual quality degrade most consistently, while audio dimensions fluctuate more variably.
  • Analysis and findings: VA-D rises from 0.025 to 0.042 and VQ-D from 0.313 to 0.361 as progressive generation extends from 60 to 180 seconds.Progressive-track drift increases with generation length.
  • Analysis and findings: Dense entity cases reduce VIF from 3.090 to 2.816 and VQ from 2.760 to 2.651, indicating visual content realization degrades with complexity.Subject appearance and background scenes remain comparatively stable.
  • Analysis and findings: Visual updates achieve less often and respond more slowly than audio updates, while history-dependent following reaches only 2.477 for adjacent and 2.321 for long-range dependencies.The reported bottlenecks concern visual changes, established-state revision, and reuse of earlier information.

5 INSIGHTS AND FUTURE DIRECTIONS

The paper frames future audio-video generation as stateful streaming: models should preserve a persistent world state, update it selectively, and render changes with low latency.

  • Maintaining long-horizon stability: Future systems should maintain perceptual quality, instruction adherence, subject appearance, and audio-video alignment throughout unbounded generation.The paper recommends jointly optimizing absolute performance and long-horizon stability rather than relying on clip-level objectives.
  • Advancing reliable interaction: Runtime updates should modify specified audio-video content while preserving valid elements and reusing relevant information from earlier interactions.Selective state-update mechanisms are proposed to jointly address update achievement, state retention, reuse, and response latency.
  • Developing joint audio-video streaming: Native joint models should generate audio and video incrementally from a shared, continuously updated latent state, alongside caching and coordinated scheduling.These architectural and system-level choices target lower latency while maintaining synchronization.
  • Developing joint audio-video streaming: The proposed directions move the field from discrete clip generation toward continuously evolving audio-video world models that can be maintained, updated, and interacted with in real time.

6 CONCLUSION

StreamAV-Bench is presented as a standardized testbed for long-horizon and interactive streaming audio-video generation, exposing limitations missed by existing benchmarks.

  • 320 expert-verified scenarios and 32 dimensions cover continuous generation, runtime updates, quality, alignment, continuity, efficiency, adherence, stability, response, and state reuse.The benchmark evaluates 13 native joint systems and cascaded pipelines under one protocol.

STREAMAV-BENCH: A COMPREHENSIVE BENCHMARK FOR STREAMING AUDIO-VIDEO GENERATION SUPPLEMENTARY MATERIAL

The supplementary material identifies the benchmark authors and describes two equally sized tracks producing 180-second outputs.

  • The author affiliations span BAAI, PKU, Kling, THU, and USTC.
  • Each track contains 160 scenarios and produces 180-second audio-video outputs from a single rollout.The progressive track uses one global-state prompt, while the interactive track uses an ordered prompt sequence.

A.2 BENCHMARK CONSTRUCTION

StreamAV-Bench constructs expert-verified progressive and interactive scenarios from structured taxonomies, scripts, prompts, and checklists. The design balances semantic coverage with content and update complexity across 320 scenarios.

  • Taxonomy design and scenario allocation: 160 scenario themes are used exactly once in each track, with allocations covering scene, audio, subject, style, entity, activity, and audio complexity.The two track instances of each theme share semantic scope but are independently realized under their temporal protocols.
  • Script construction and interval annotation: Annotators create structured scripts describing visual and audio content, with progressive scripts spanning 180-second rollouts and interactive scripts specifying an initial state plus five updates.Interactive scripts jointly plan the initial world and runtime updates as a single state-consistent sequence.
  • Script construction and interval annotation: Interactive updates are annotated by interaction type, modality, and temporal dependency, while requiring causal ordering and persistence of unchanged states.Dependencies may be adjacent or long-range, and established subjects, objects, activities, sounds, and states persist until explicitly changed.
  • Prompt generation: GPT-5.4 converts structured scripts into one global progressive prompt or an initial prompt followed by five concise interactive updates.The prompts follow a common generation-oriented style.
  • Checklist generation and expert verification: Two independent domain experts and a third adjudicator verify taxonomy consistency, temporal dependencies, feasibility, content coverage, and audio-video plausibility.They also assess temporal and cross-update state consistency and evaluation consistency.
  • Taxonomy design and scenario allocation: The benchmark contains 320 scenarios spanning 8 scene domains, 5 audio domains, 5 subject categories, and 4 visual styles.The set includes 192 photorealistic and 128 non-photorealistic scenarios.

B EVALUATION FRAMEWORK DETAILS

The evaluation framework organizes 32 metrics across progressive, interactive, and shared dimensions. It combines automated, expert-model, and MLLM assessments to measure quality, adherence, stability, continuity, efficiency, response, and state reuse.

  • Framework organization: The 32 metrics are organized into six categories spanning progressive instruction adherence and stability, interactive response and state reuse, and shared continuity, efficiency, quality, and alignment.Scores are generally averaged within cases and then uniformly across cases.
  • Shared quality and alignment: Visual quality, audio quality, and cross-modal alignment are evaluated over six non-overlapping 30-second intervals.The category includes visual aesthetics, visual and production quality, audio quality, alignment, and synchronization metrics.
  • Streaming continuity and efficiency: Native Boundary Continuity scores black frames, repeated frames, freezing, and flashes at model-native generation-unit boundaries using NBC = clip(100 −(70rb + 45rd + 12tf + 8nf), 0, 100).Scores are averaged over boundaries and cases from both tracks; higher is better.
  • Progressive evaluation: Progressive metrics compare instruction adherence and endpoint drift across the full generation horizon, including visual, audio, quality, subject, background, and audio-video drift.Drift metrics compare the first and last 30-second intervals, with lower values preferred.
  • Interactive evaluation: Interactive metrics evaluate update fulfillment, continuity, achievement rates, response latencies, and history-dependent state retention at five runtime boundaries.Response latencies are conditioned on achieved targets, while achievement rates count updates with clearly realized core targets.
  • Interactive evaluation: History-dependency following measures whether systems preserve unchanged states and reuse earlier interaction information when the required source state is established.HDF is computed only when its required source state is present.

C IMPLEMENTATION DETAILS

The study evaluates 13 systems under standardized progressive and interactive protocols, then analyzes temporal degradation and success-adjusted interactive behavior. Results show dimension-specific drift and complementary trade-offs between achievement, latency, and state dependency.

  • Evaluated systems: The evaluation covers 13 systems across native joint streaming audio-video models and cascaded pipelines pairing HunyuanVideo-Foley with streaming video generators.All systems are compared under the benchmark’s two generation paradigms.
  • Evaluation protocol: All systems receive identical inputs: one global prompt for progressive generation and an ordered prompt sequence with fixed 30-second interactive updates.Outputs are converted to a common audio-video format before evaluation.
  • Initial-world alignment: HappyOyster leads Initial VIF while SWIFT leads Initial AIF, so no single system leads both modalities in establishing the starting world.The initial-world evaluation applies visual and audio instruction-fulfillment criteria to the first interval.
  • Interactive response: HappyOyster achieves the highest PVUAR and PAUAR, whereas SWIFT and OmniForcing attain lower visual or audio response latencies over smaller successful subsets.The failure-penalized comparison jointly evaluates achievement and latency.
  • State retention and reuse: HappyOyster leads adjacent source establishment and both conditional HDF scores, while SWIFT leads long-range source establishment.OmniForcing’s ranking shows that source establishment and dependency following capture distinct failure modes.
  • Success- and coverage-adjusted diagnostics: Success-adjusted latencies penalize unachieved updates, and coverage-adjusted HDF scores combine source establishment with conditional dependency quality.These diagnostics assign fixed penalties or incorporate coverage rather than excluding unsuccessful cases.
  • Success- and coverage-adjusted diagnostics: HappyOyster has the lowest SA-PVRL and highest CA-HDF-A and CA-HDF-L, while PixVerse R1 has the lowest SA-PARL.The adjusted metrics change the interpretation of latency and dependency quality by accounting for unsuccessful or unavailable cases.
  • Temporal evolution: VA, AVAlign, and VQ degrade in 13, 12, and 10 of 13 systems, while PQ, AQ, and AVSync show mixed directions.Trajectory variation reveals instability even when some dimensions have improving overall trends.

D.6 CONTENT AND UPDATE COMPLEXITY RESULTS

Content complexity primarily degrades visual realization in progressive generation, while interactive updates expose bottlenecks in visual changes, state revision, and history reuse.

  • Progressive track: VIF decreases from 3.090 to 2.816 and VQ from 2.760 to 2.651 as entity density increases from single-entity to dense cases.Entity density produces the clearest visual degradation, while subject appearance and background scenes remain comparatively stable.
  • Progressive track: VQ decreases by 0.129 from atomic to concurrent activities without a corresponding decline in audio instruction fulfillment.
  • Interactive track: Visual and audio update fulfillment are comparable at 2.347 and 2.332, respectively, but visual updates achieve less and respond more slowly.PVUAR/PAUAR are 0.243/0.331, while PVRL/PARL are 13.534/12.437 seconds.
  • Interactive track: Direct agent actions attain higher VUF than entity-state and environment changes, at 2.418 versus 2.274 and 2.283.
  • Interactive track: History-dependent fulfillment reaches 2.477 for adjacent dependencies and 2.321 for long-range dependencies.The reported interaction bottlenecks concern reliably realizing visual changes, revising established states, and reusing earlier information.

E HUMAN ALIGNMENT

Human alignment is assessed through blind pairwise comparisons, while checklist templates operationalize visual and audio instruction metrics and runtime update decomposition. The selected metrics show strong model-level agreement with human judgments.

  • Human evaluation: Blind pairwise comparisons evaluate 60 balanced cases across both tracks and all 13 systems, using three annotators per comparison with randomized side order and ties allowed.Shared metrics use 30 cases per track, while track-specific metrics use the corresponding 30-case subset.
  • Human alignment: All 17 metrics show strong model-level human alignment, with Spearman correlations ranging from 0.77 to 0.94.Prompt-based visual continuity has the highest correlation, while response latencies correlate less strongly than achievement and fulfillment metrics.
  • Checklist construction: The visual checklist organizes applicable prompt requirements into scene, visual style, subjects, and activity fields.Each field becomes a concise, self-contained Yes/No question using only explicitly stated visual requirements.
  • Checklist construction: The audio checklist covers sound source, sound content, and temporal relation, with questions restricted to explicitly stated auditory requirements.Fields may be null when no applicable requirement exists, and related requirements within a field may be combined.
  • Update decomposition: The joint runtime-update splitter separates explicitly requested visual and audio changes while preserving original wording and semantic details where possible.
  • Evaluation cases: Progressive and interactive examples illustrate continuous prompts versus temporally dependent prompt schedules, including later updates responding to earlier changes.The examples cover a 180-second sailing sequence and a bread-baking process with linked state updates.
Loading 2608.26336v1…