Source-linked AI summary
StreamEval: Evaluating Interactive Streaming Video Generation Models
StreamEval Team
TL;DR
Interactive streaming video requires systems to respond to incoming requests while preserving relevant context and state. StreamEval benchmarks this capability across five tasks and evaluates 37 models, finding complementary strengths across tasks while request execution and reference-edit completion remain challenging.
Problem
Interactive video systems must respond to incoming requests as context evolves, while preserving relevant prior content and state throughout a session.
Source: §1 · p. 3Method
StreamEval evaluates five interactive streaming tasks using 300 scripted sessions with timed multimodal inputs, explicit change and preservation criteria, and ten task-specific metrics across four dimensions.
Source: §1 · p. 3Results
Models show complementary strengths across tasks, while reference-editing Edit Success remains low, with the highest reported score at 15.00 on a 0–100 scale.
Source: §1 · p. 3, §6 · p. 16Takeaways & Limitations
Reliable request execution under timing and preservation constraints remains a central challenge for advancing omni-modal real-time interactive systems.
Source: §6 · p. 16The benchmark uses synthetic portraits and speech, controlled acoustic variations, and licensed source videos; broader identities and environments remain future extensions.
Source: §4.2 · p. 13
Abstract
from arXiv · showhide
As streaming video generation advances, enabling real-time interaction with generated video is becoming increasingly important. We introduce StreamEval, a unified benchmark for interactive streaming video generation spanning five tasks: avatar dialogue, avatar performance, world generation, instruction editing, and reference editing. StreamEval comprises 300 scripted test cases lasting 15-300s, each specifying initial assets, incrementally arriving multimodal requests, and explicit requirements for state changes and content preservation. Performance is assessed along four evaluation dimensions: task execution, state consistency, visual fidelity, and audiovisual coordination, instantiated through ten task-specific metrics on a 0-100 scale. Evaluating 37 distinct models across 300 test cases, we find that high visual quality and identity retention do not necessarily coincide with reliable request execution or sustained state consistency. In particular, request execution remains challenging: the highest task-specific scores are 36.90 for Following among seven avatar-performance models, 25.93 for Following in text-conditioned world generation, and 15.00 for Edit Success among three reported reference-editing models. These findings identify reliable request execution under timing and preservation constraints as a central challenge for advancing omni-modal real-time interactive systems. The leaderboard is available at https://streameval.ai
1 Introduction
StreamEval frames interactive streaming video as a problem of executing timed requests while preserving relevant context and state. It unifies five tasks and evaluates 37 models, finding distinct strengths across tasks alongside persistent challenges in request execution.
Motivation
- Interactive systems must execute requests as user inputs and context evolve, rather than produce only plausible individual frames or clips.Requests carry timing and scope requirements, so evaluation must assess execution and state consistency alongside visual fidelity and audiovisual coordination.Source: §1 · p. 3
Relation to prior work
- Existing benchmarks cover complementary aspects, while StreamEval unifies avatar interaction, world generation, and real-time editing with explicit task-specific execution and preservation criteria.VideoFDB focuses on conversational dynamics, complementing StreamEval’s task-directed evaluation.Source: §1 · p. 3
Benchmark
- The benchmark covers digital-human dialogue, avatar performance, world generation, instruction editing, and reference editing in 300 scripted sessions lasting 15–300 seconds.Each session specifies initial assets, timed requests, and criteria for what should change and remain preserved.Source: §1 · p. 3
Results
- Vidu S2 Avatar leads avatar performance Overall, MemFlow leads world-generation Overall, and Vidu S2 Editing leads both instruction- and reference-editing Overall.LiveTalk leads avatar-performance Quality, while JoyAI-Video-Edit leads instruction-editing Preservation.Source: §1 · p. 3
Contributions
- StreamEval evaluates 37 models across 300 test cases and five tasks, revealing limited instruction following in avatar performance and world generation and low full-edit completion in reference editing.Its contribution combines timed multimodal requests with explicit execution and preservation requirements and ten task-specific metrics across four dimensions.Source: §1 · p. 3
2 The StreamEval Benchmark
StreamEval tests interactive video generation across five tasks with timed inputs and explicit requirements for execution and preservation. Its benchmark spans multiple durations and interaction types, and is designed to assess task coverage, state retention, and content quality.
2.1 Motivation
- StreamEval treats real-time interaction as executing evolving requests while preserving relevant context and state across avatar interaction, world generation, and video editing.A scheduler releases inputs incrementally and advances editing sources on the session clock; input events specify control timing, while scoring opportunities specify what to accomplish or retain.Source: §2.1 · p. 4
2.2 Benchmark Construction
- The benchmark tests five tasks through scripted sessions with explicit goals, timed controls, protected attributes, and checkpoint outcomes, including successive edits and restoration.It covers individual operations, combined requests, recovery, and more demanding interactions such as repair and multi-target updates.Source: §2.2 · pp. 4–5
2.3 Data Statistics
- The 300 cases span five durations from 15 to 300 seconds, contain 1,347 planned online events, and receive equal aggregation weight across duration strata.Each task has 60 cases, balanced across durations; a pass through the cases represents 525 minutes of active streaming generation.Source: §2.3 · p. 5
2.4 Comparison with Existing Benchmarks
- StreamEval balances five tasks across 15–300-second horizons, extending editing cases beyond LongV2VBench and world-generation cases beyond StreamAV-Bench rollouts.Its distinction is balanced coverage across tasks and durations with incrementally released multimodal controls and continuously advancing editing sources.Source: §2.4 · p. 6
- The shared framework assesses 37 models across application domains for request execution and state preservation alongside content quality.Prior benchmarks differ in task coverage and interaction protocols, which depend jointly on task interfaces and control timing.Source: §2.4 · pp. 5–6
2.5 Quality Verification
- Benchmark checks verify media integrity, timing bounds, and agreement between events and scoring requirements; speech transcripts must meet a normalized word-error rate of at most 0.25.Targeted probes assess production judges against false-positive, false-negative, and unresolved-decision thresholds.Source: §2.5 · p. 6
3 System Evaluation
The evaluation examines how systems handle evolving requests, preserve relevant state, and render video across five tasks.
Evaluation
- The evaluation measures how systems execute evolving requests across the five tasks.Each task and input setting has a defined metric profile.Source: §3 · p. 7
- The evaluation assesses whether systems retain relevant state while responding to requests.Each task has an Overall score.Source: §3 · p. 7
- The evaluation also considers the rendered video produced by systems across the five tasks.Metric profiles vary by task and input setting.Source: §3 · p. 7
3.1 Evaluation Setup
The setup evaluates 37 distinct models in 47 task configurations across five tasks, using 300 cases, repeated runs, and task-specific metrics.
Evaluated models
- The evaluation reports 37 distinct models across 47 task configurations, with editing tables limited to configurations whose Overall score is at least 5.Fixed-avatar systems are rated against their own opening frames, and Identity contributes to their Overall score.Source: §3.1 · p. 7
Evaluation tasks
- The five tasks cover 15 dialogue, 7 avatar-performance, 17 world-generation, 5 instruction-editing, and 3 reference-editing configurations.The evaluation uses 300 cases, three repeats per case and model, and five equally weighted duration strata from 15 to 300 seconds.Source: §3.1 · p. 8
Evaluation metrics
- Ten metrics assess task execution, state consistency, visual fidelity, and audiovisual coordination, with each task reporting an equal-weight mean of its applicable metrics as Quality-Overall.Avatar tasks additionally report a customization bonus and its sum with Quality-Overall as Overall.Source: §3.1 · p. 8
3.2 Metrics and Scoring
The metrics distinguish task execution, state consistency, visual fidelity, and audiovisual coordination. Task-specific scores aggregate repeats, cases, and duration strata, with an avatar-customization bonus applied separately.
3.2.1 Task-Specific Metrics
- Quality scores clip defects, while Stability measures qualifying duration; because Stability is gated by Quality, the metrics overlap.A high Stability score requires temporal consistency and acceptable visual quality.Source: §3.2.1 · p. 8
Digital-human dialogue.
- Dialogue scores planned turns meeting answer, content, conversational, and timing requirements; Identity rates appearance retention, and A–V Sync checks required speech events.Identity allows authorized expression, pose, crop, and lighting changes; sparse source sampling limits A–V Sync’s temporal resolution.Source: Digital-human dialogue · p. 8
Avatar performance.
- Following measures requested avatar actions on specified targets; in world generation, Following checks scene operations and Continuity measures duration meeting motion and scene requirements.The world-generation metrics apply in both text-conditioned and image-conditioned settings.Source: Avatar performance · p. 8
Instruction editing.
- Instruction-editing metrics distinguish requested operation execution, full edit completion, protected-content preservation, and visual fidelity.Edit Success checks scope, active state, deadline, persistence, and protected content; Preservation includes restoration against the contemporaneous source.Source: Instruction editing · p. 8
Reference editing.
- Reference measures declared appearance attributes while the supplied image is active; Identity instead compares against an initialization portrait or opening frame.Edit Success checks the full editing contract, while Preservation assesses protected source content and motion.Source: Reference editing · p. 9
3.2.2 Score Aggregation
- Scores average three repeats, then cases within each duration stratum, then five duration strata; task-specific Quality-Overall equally averages applicable metric aggregates.Avatar tasks add +5 for custom-avatar support; other systems receive 0, and the bonus does not alter underlying metrics. Figure 4 illustrates recorded inputs and outputs and the quality, stability, reference, and preservation checks.Source: §3.2.2 · pp. 9–10, Figure 4 · p. 9
3.3 Experimental Results
Across five streaming tasks, the results show distinct model strengths, while request execution remains a challenge even when visual quality or state-related scores are high.
3.3.1 Digital-Human Dialogue
- Vidu S2 Avatar leads custom-avatar dialogue with 93.98 Overall, while Tavus CVI Full leads A–V Sync at 94.44 in the fixed-avatar setting.MuseTalk 1.5 pairs 95.34 Quality and 97.11 Stability with 38.09 A–V Sync and 54.34 Dialogue.Source: §3.3.1 · p. 10, Table 2 · p. 10
3.3 Experimental Results
- Following reaches only 36.90 for Vidu S2 Avatar in avatar performance, while reference-editing Edit Success peaks at 15.00 for Vidu S2 Editing across three reported systems.In text-conditioned world generation, Krea Realtime leads Following at 25.93.Source: §3.3.2 · p. 11, Table 3 · p. 11, §3.3.5 · p. 12
3.3.2 Avatar Performance
- Vidu S2 Avatar leads avatar performance at 81.71 Overall and 36.90 Following, while LiveTalk 1.3B leads Quality at 71.27.Following ranges from 6.37 for Avatar-Forever 22B to 36.90 for Vidu S2 Avatar, including systems above 80 Identity and 96 Stability.Source: §3.3.2 · p. 11, Table 3 · p. 11
3.3.3 World Generation
- MemFlow leads world generation at 33.36 Overall, but Krea Realtime leads Following at 25.93 and Matrix-Game 3.5 leads Continuity at 34.50.HappyOyster 1.0 Adventure leads Quality (58.34) and Stability (56.67), but scores 5.56 in Following and 2.78 in Continuity.Source: §3.3.3 · p. 11, Table 4 · p. 12
3.3.4 Instruction Editing
- Vidu S2 Editing leads instruction editing at 45.43 Overall, while JoyAI-Video-Edit leads Preservation at 83.33.For Vidu S2 Editing, Following is 72.10 and Edit Success is 57.20; Quality and Stability remain low even among the highest-scoring systems.Source: §3.3.4 · pp. 11–12, Table 5 · p. 12
3.3.5 Reference Editing
- Vidu S2 Editing leads reference editing at 39.02 Overall and 15.00 Edit Success, but completion remains challenging across all three reported systems.JoyAI-Video-Edit and XMax X2.0 score above 20 in Reference but only 8.89 and 1.11 in Edit Success, respectively.Source: §3.3.5 · p. 12, Table 6 · p. 13
4 Analysis and Qualitative Results
A recorded reference-editing session illustrates how source preservation and adopting a changing reference can diverge; the benchmark also identifies its media coverage and possible extensions.
4 Analysis and Qualitative Results
- The recorded session is presented to illustrate divergence between source preservation and reference adoption, alongside a summary of benchmark coverage and evaluation scope.Source: §4 · p. 13, §4.1 · p. 13, §4.2 · p. 13
4.1 A Recorded Reference-Editing Session
- In the recorded session, both sampled outputs preserve the underlying scene but retain the grey cape instead of adopting the requested gold satin appearance.Reference A is preloaded, Reference B arrives at 10 s, and removal is requested at 20 s.Source: §4.1 · p. 13, Figure 5 · p. 13
4.2 Benchmark Coverage
- The benchmark combines synthetic portraits and speech, controlled acoustic variations, and licensed source videos.Possible extensions include broader identities and environments, global resets, and selective restoration while concurrent edits remain active.Source: §4.2 · p. 13
5 Related Work
Related work covers efficient streaming generation, changing controls, streaming editing, and benchmarks for interactive behavior. StreamEval brings these evaluation requirements together across five tasks and four assessment dimensions.
5.1.1 Streaming Video Generation
5.1.2 Streaming Video Editing
- Streaming-editing work develops incremental source-conditioned editing and interfaces for successive instructions or references that specify changes and retained content.Examples include StreamV2V [35], LiveEdit [55], and Vidu S2-Editing [79].Source: §5.1.2 · p. 15
5.2 Evaluating Streaming Video Systems
- Prior evaluation benchmarks address content quality, temporal change and memory, multi-turn interaction, avatar behavior, and source-preserving video editing.StreamEval links five tasks to timed inputs, expected changes, retained state or source content, and evaluation of task execution, state consistency, visual fidelity, and audiovisual coordination.Source: §5.2 · pp. 15–16
6 Conclusion
StreamEval unifies five interactive streaming video tasks in 300 scripted cases with explicit request-execution and state-preservation requirements. Across 37 models, online action following and reference-based editing remain challenging.
6 Conclusion
- StreamEval evaluates 37 models across five tasks and 300 scripted cases; the highest reference-editing Edit Success score is 15.00 on a 0–100 scale.The conclusion identifies improving request execution alongside visual fidelity and state continuity as an important direction for model design.Source: §6 · p. 16