UniVA: Universal Video Agent towards Open-Source Next-Generation Video Generalist

1 Introduction

• Specialized video tools and models operate in isolation, leaving complex multi-step goals to brittle, labor-intensive manual coordination.The introduction identifies the lack of a unified system for reasoning across multiple video tasks and steps as a bottleneck.Source: §1 · pp. 1–2
• UniVA-Bench evaluates multi-step video tasks across understanding, segmentation, editing, and generation, including planning quality, memory use, and tool-routing efficiency.The benchmark uses goal cards with gold artifacts and measures compositionality, tool swaps, and long-form reasoning.Source: §1 · p. 3
• UniVA unifies video understanding, editing, and generation in an open-source framework with a Plan–Act architecture and hierarchical memory for long-horizon goals and user intent.The introduction describes this design as enabling automated, interactive, and proactive video creation.Source: §1 · p. 3
• Its MCP-based tool ecosystem connects open-source and API-based modules in a plug-and-play framework for any-conditioned generation and cross-modal editing.The tool hub includes video, non-video, and non-AI tools and is designed for extension with new capabilities.Source: §1 · p. 3

Video Processing and Intelligence

• Video intelligence spans understanding, fine-grained segmentation, conditional generation, and semantic editing, but most systems remain fragmented and task-specific.Prior work covers action recognition, event detection, captioning, retrieval, question answering, pixel-level tasks, and video synthesis.Source: §2 · pp. 3–4

Agents for Video Intelligence

• UniVA’s Plan–Act architecture decomposes text, image, or video input into subtasks and coordinates execution through task-specific memory and MCP servers.The system coordinates video, AI, and non-AI tools and can produce text, image, video, and audio outputs.Source: Figure 2 · p. 4

Toward Unified Video Modeling

• Unified video models combine understanding and generation or add segmentation, but static pipelines make them difficult to schedule, coordinate, and extend.The related work identifies dynamic orchestration of heterogeneous modules as a need.Source: §2 · p. 4

Agents for Video Intelligence

• Video agents use planning, interaction, and memory across tasks, while communication and coordination in multi-agent systems remain open challenges.Prior applications include reasoning, editing, stylization, and story generation.Source: §2 · p. 4

3.1 Problem Formulation

• The problem is to choose actions that transform an initial state, including user-provided media, into a final state satisfying the user’s goal.The desired action sequence balances broad tool use with complex, synergistic actions and a high-quality final state.Source: §3.1 · pp. 4–5

3.2 The UniVA Control Core

• The control core pairs a decision engine that formulates the policy with memory that manages the current state and interaction history.Persistent and task-specific memory support the strategy and ongoing workflow, respectively.Source: §3.2 · p. 5, §3.2.2 · p. 5

3.2.1 Plan–Act Dual Agent Architecture

• The Planner maps the goal and current state to an abstract plan, while the Actor converts each plan step into concrete tool actions.After each tool call, the Actor returns its output to the Planner; the separation keeps planning strategic and tool use focused on execution.Source: §3.2.1 · p. 5

3.2.2 Memory Mechanism

• Global memory stores reusable knowledge, task memory retains workflow artifacts and traces, and user memory tracks preferences and interaction history.Task memory enables later steps to reuse prior outputs without recomputation, while global and user memory support cross-task context and adaptive behavior.Source: §3.2.2 · p. 5

3.3 Tools Interaction

• The MCP gateway registers available functions, validates and executes calls, records outputs, and allows tools to be added or replaced without changing the Planner or Actor.UniVA’s action space is designed to be open and extensible.Source: §3.3 · pp. 5–6

3.4 Framework Walkthrough

• In the framework walkthrough, the Planner formulates a plan from the current state and goal, the Actor executes its actions, and memory records outputs for subsequent steps.The examples contrast a multi-step generation workflow with interactive editing that chains diverse tools for a context-dependent outcome.Source: §3.4 · p. 6, Figure 3 · p. 6

3.5 System Implementation

• The interactive web editor reflects tool-call results on its timeline and preview canvas, letting users move between AI-driven creation and hands-on editing.The agent parses requests, formulates a plan, and executes the required tool calls in the background.Source: §3.5 · p. 6, Figure 4 · p. 7

4.1 Benchmark Definition

• UniVA-Bench shifts evaluation from isolated tasks to end-to-end tool-augmented workflows, pairing functional-module performance with agentic probes of planning, dependencies, replanning, and memory.Source: §4.1 · p. 7

4.2 Evaluation Tasks

• Tasks cover long-video QA; text-, image/entity-, and video-conditioned generation; multi-step consistent editing; and occlusion-heavy segmentation, with Figure 6 showing representative category examples.Source: §4.2 · pp. 7–8, Figure 6 · p. 9
• Agentic probing uses 50 storyboard-to-intent planning instances and expert-referenced pipelines to assess wPED, DepCov, and ReplanQ, including injected failures and memory analyses.Source: §B.1 · p. 33

4.3 Evaluation Protocol

• Evaluation combines task-specific quality, MLLM pairwise user preference, and planning metrics; reported measures include CLIP, DINO, segmentation J/F, normalized QA accuracy, wPED, DepCov, and ReplanQ.Source: §4.3 · pp. 8–9

5.1.1 Generation

• LongText2Video reached CLIP Score 0.2814 and MLLM Judge score 3.333; Video2Video scored 4.068 on MLLM Judge, while Seedance showed strong DINO Score performance in Entities2Video.UniVA remained competitive on Entities2Video subject consistency.Source: §5.1.1 · p. 10, Table 1 · p. 10

5.1.2 Understanding

• 0.76 was UniVA’s highest LongVideo Understanding accuracy, exceeding the compared single-inference models.The authors attribute this to decomposing the video and complex query into manageable subtasks.Source: §5.1.2 · p. 10, Table 2 · p. 10

5.1.3 Editing

• UniVA grounds long-video editing objects in persistent semantic context established through its integrated Understanding module and Probing tool.This context supports editing across long-term, cross-shot video.Source: §5.1.3 · p. 10, Table 2 · p. 10

5.1.4 Segmentation

• UniVA outperformed the best scores on all segmentation metrics, with the authors attributing the advantage to resolving ambiguities through the co-located Understanding module.The Probing tool can use narrative context to help determine whether an occluded object is the same entity when it reappears.Source: Table 2 · p. 10, §5.1.4 · p. 11

5.2.1 Planning Capability

• Plan-Act more than doubled valid-plan Success Rate (45.0% vs. 20.0%) and more than doubled wPED (0.117 versus 0.050).Claude-Sonnet-4 was selected as Planner after outperforming two other LLMs in DepCov and ReplanQ.Source: §5.2.1 · p. 11, Figure 8 · p. 11, Figure 7 · p. 11

5.2.2 Memory Capability

• Global memory generally raised wPED and prevented many zero-score planning failures; user memory improved intent alignment, while storyboards boosted all quality metrics.The authors connect storyboards’ intermediate representation of the creative goal with semantic coherence and cross-shot consistency.Source: §5.2.2 · pp. 11–12, Figure 10 · p. 12, Figure 11 · p. 12

5.3 Human Evaluation

• UniVA achieved the highest human preference scores in four of five evaluated dimensions, aligning with the automated metric patterns.The study compared UniVA and baseline generation results across LongText2Video, Image2Video, and Video2Video.Source: §5.3 · p. 12, Figure 12 · p. 12

5.4 Qualitative Case Studies

• Case studies show proactive multi-round creation, any-conditioned inputs, long-instruction following, multi-camera narratives, and workflows spanning understanding, editing, and generation.Examples include maintaining character consistency in long videos and generating coherent multi-camera transitions.Source: §5.4 · pp. 13–15, Figure 19 · p. 14, Figure 20 · p. 14

6 Conclusion

• UniVA’s experiments show competitive performance across video tasks and describe “Agentic Synergy” as solving consistency problems through dynamic information flow between tools.The conclusion frames UniVA as an integrated system rather than a collection of tools.Source: §6 · p. 16
• The authors argue that video intelligence should move beyond isolated single-task models toward integrated systems for complex, collaborative workflows.They present UniVA’s platform, its synergistic capabilities, and UniVA-Bench as contributions.Source: §6 · p. 16

Appendix

• The appendix includes detailed methodology and experiment settings.It lists supplementary sections A and C.Source: Appendix · p. 20

A.1 Principles

• UniVA combines plug-and-play MCP modules with separated planning and execution, proactive goal-checking, hierarchical memory, and atomic operations assembled freely or as robust workflows.The Planner can assess intermediate results and revise its plan; global, task-specific, and user memory support continuity across extended interactions.Source: §A.1 · p. 20

A.2 MCP Function Walkthrough

• UniVA’s modular MCP toolset groups functions into video, non-video, and non-AI categories, with each function classified as an atomic operation or a composed workflow.The taxonomy distinguishes single-purpose tools from higher-level functions that combine tools for multi-step tasks.Source: §A.2 · p. 21, Figure 24 · p. 22

A.2.1 Video Tools

• Video tools cover editing, generation, tracking, and understanding, from object and depth edits to prompt-conditioned video creation, segmentation, and analysis.The listed tools include short text- or image-conditioned generation, video extension, and end-to-end story-video workflows.Source: §A.2.1 · pp. 21, 23

A.2.2 Non-Video Tools

• Non-video tools provide audio creation and processing, including foley, speech, transcription, voice cloning, and music, as well as image generation and editing.Image generation can be conditioned on an input image for style or identity consistency.Source: §A.2.2 · p. 23

A.2.3 Non-AI Tools

• Non-AI utilities merge clips, add transitions or subtitles, and search royalty-free media; MCP modularity lets functions be registered, replaced, or expanded without changing the core architecture.The framework is presented as an open, evolving ecosystem that can incorporate new video models, APIs, or custom tools.Source: §A.2.3 · pp. 23–24

A.3 Core Prompt

• The core planner prompt defines Univideo as a video-generation and processing planner that turns user requests into detailed, stepwise plans and selects tools for each operation.The paper says it will open-source the code resources, including agent prompts.Source: §A.3 · p. 24

I. AVAILABLE TOOLS OVERVIEW

• The tool overview distinguishes independent, low-level atomic functions from workflows that execute a defined sequence through a single interface.Examples include story-video workflows that create storyboards, characters, keyframes, and video segments before merging the result.Source: §I · p. 24
• The available tools include text-prompted segmentation and image generation from text or a reference image.Video segmentation can localize and track prompted objects or generate masks for all detectable objects; image-to-image generation retains consistency with the input image’s characters or style.Source: §I · pp. 25–26

II. CORE PLANNING LOGIC & TOOL SELECTION GUIDELINES

• The planner uses vision2text_gen when visual semantics are absent, generates missing materials before later steps, and manages references for style consistency.Source: §II · p. 26

III. PLAN OUTPUT FORMAT

• Plans must list numbered actions, tools, parameters, input paths, dependencies, statuses, and outputs, with separate steps for separate material inputs.The specified format is structured JSON; the first step is ongoing and unexecuted steps are pending.Source: §III · p. 26, §IV · p. 27

IV. PLANNING UPDATES

• Plans update sequentially after each step’s result, keep exactly one step ongoing, and adjust dynamically when a step fails.Once the plan succeeds, no further updates are needed beyond a short summary.Source: §IV · p. 27

I. Tool Calling Protocol:

• Tools are called one at a time in standardized MCP format, with each call informed by prior results and failures prompting analysis and adjustment.Source: §I · pp. 27–28

1. Plan Reception and Video Task Understanding

• The video executor receives detailed plans and accounts for 5-second segments, cross-segment visual consistency, and dependencies on clips and references.Source: §1 · p. 28

2. Video-Centric Step-by-Step Execution

• Video actuation requires checking each step’s parameters before calling tools; the example enforces exactly 5-second MP4 segments.The executor is also assigned the role of turning a one-sentence video concept into a detailed storyboard for downstream generation.Source: §2 · pp. 28–29

1. Input

• The input is exactly one short sentence summarizing the video’s storyline or theme, which the storyboard must follow without alteration.The storyboard is to be based solely on that sentence.Source: §1 · p. 29

2. Output

• The output is valid JSON with characters, an ordered shots array, and an overall style description.Character entries have unique IDs, names, and static appearance descriptions; each shot includes its duration, setting, action, characters, composition, audio, and suggested video-generation type.Source: §2 · pp. 29–30

6. General Script Generation Rules

• When no duration is specified, the storyboard assumes 60 seconds and distributes that time across shots; shot count should fit the storyline and transitions should flow smoothly.The prompt gives roughly 12–15 shots for a one-minute video and 30–40 for three minutes, adjusting as needed.Source: §2 · p. 29, §6 · p. 31
• Shot descriptions separate static composition from one dynamic action, and character references use IDs from the characters array.The static description covers poses and placement; plot correspondence describes exactly one action per shot.Source: §6 · p. 31

Understanding (Long-Video QA)

• Long-video QA uses 10 sampled Video-MME videos, with 10 questions per video answered in a single inference.Questions are generated with Gemini 2.5 Pro and draw on perspectives shown in Table 3.Source: §B.1 · p. 33

Generation

• Generation data include long, noisy prompts for LongText2Video and rewritten prompts for 10 sampled Opens2v-nexus examples in Image/Entities2Video.LongText2Video prompts are created from GPT-generated storyboards.Source: §B.1 · p. 33
• Video2Video evaluation separates story alignment, style alignment, and semantic alignment, using 10 SF20k videos per task with manually generated prompts.The settings respectively alter style, storyline, or both while preserving the specified remaining video elements.Source: §B.1 · p. 33

Editing (Long Video)

• Long-video editing uses 10 SF20k videos with prompts manually curated from each video's content.Source: §B.1 · p. 33

Segmentation (Long Video)

• Long-video segmentation uses 10 instances made by randomly concatenating DAVIS2017 clips, covering diverse scenes and occlusions.Source: §B.1 · p. 33

Agentic probing sets

• Agentic probes compare Single-Agent with Plan-Act on 50 storyboard-to-user-intent cases and assess planning metrics under injected failures using expert references.Memory analyses cover historical trajectories, user preferences, and task context such as storyboards.Source: §B.1 · p. 33

B.2.1 Subject metrics (task quality).

• CLIP Score measures instruction-to-video alignment, DINO Score measures reference-entity identity stability, segmentation uses J/F/mIoU, and Understanding Score is normalized QA accuracy.Higher values are better for CLIP, DINO, and segmentation metrics; the understanding questions cover semantics and aesthetics.Source: §B.2.1 · p. 34

B.2.2 MLLM as a judge (preference).

• Pairwise preferences come from open- and closed-source judges, with majority voting, discarded ties, and significance tests when applicable.Judges receive instructions, relevant references, and debiased captions; a standardized prompt template requests structured output.Source: §B.2.2 · p. 34
• The MLLM rubric scores semantic content and object-spatial relations, using timestamped evidence to assess correctness and stability.The rubric includes object and scene presence, count, arrangement, occlusion, and relative spatial relations.Source: §B.2.2 · pp. 34–35
• The rubric separately scores specified action continuity and attribute fidelity, including whether colors and other attributes remain stable over time.Source: §B.2.2 · p. 35
• Style consistency evaluates appearance and camera movement, while overall video-text consistency checks holistic alignment without double-counting finer issues.The overall dimension is null when no text prompt is present.Source: §B.2.2 · p. 35

• wPED compares the tool-name sequence in a generated plan with an expert plan using normalized, inverted Levenshtein edit distance.A score closer to 1.0 indicates closer structural alignment.Source: §B.2.3 · p. 36
• DepCov measures the fraction of rule-based workflow dependencies satisfied by a generated plan, such as generating content before editing it.A higher score indicates greater adherence to procedural constraints.Source: §B.2.3 · p. 36
• ReplanQ compares the original and revised plan suffixes from the failure point using normalized, inverted Levenshtein distance; higher scores indicate more efficient recovery.The metric rewards minimal plan modifications after a simulated execution failure.Source: §B.2.3 · p. 36

C.1 UniVA’s Configuration

• Table 4 is identified as the table of UniVA's configuration.Source: Table 4 · p. 36

C.2 Baseline Configurations

• Generation outputs are standardized to 480p and 24 fps, with preprocessing to accommodate baselines lacking multi-image or video-conditioned input support.Entities2Video reference images are composited for single-image baselines; a Qwen2.5-VL-72B caption is prepended for text-only Video2Video baselines.Source: §C.2 · p. 37
• Generation baselines include LTX-Video, Seedance, and Wan, configured with the specified public-repository settings or 480p Wavespeed API models.Source: §C.2 · p. 37
• Understanding baselines receive video at 1 fps with a maximum of 128 frames.Source: §C.2 · p. 37
• The editing baseline uses Runway Aleph on five-second clips that are merged for evaluation, while Sa2Va-4B receives the full video and segmentation prompt.Source: §C.2 · p. 37