Source-linked AI summary
CutVerse: A Compositional GUI Agents Benchmark for Media Post-Production Editing
Haobo Hu, Xiangwu Guo, Zhiheng Chen, Difei Gao, Haotian Liu, Libiao Jin, Qi Mao
TL;DR
Professional creative workflows remain underexplored because they require dense interfaces, tightly coupled multimodal operations, and reliable long-horizon execution. CutVerse introduces a benchmark, structured-trajectory parser, and diagnostic evaluation framework built from authentic media post-production workflows. Evaluations reveal a substantial gap between local procedural competence and reliable completion of complex editing tasks.
Problem
Professional media post-production requires dense interfaces, tightly coupled multimodal operations, and long-horizon coordination that existing evaluation settings do not capture.
Method
CutVerse curates authentic workflows across professional applications and parses synchronized screen recordings and low-level events into grounded, milestone-based GUI trajectories.
Results
Current agents handle structured procedural operations but struggle to sustain complex editing workflows requiring precise spatial grounding, temporal coordination, and compositional control.
Takeaways & Limitations
CutVerse provides a practical foundation for evaluating and advancing end-to-end multimedia production through authentic, high-fidelity software interaction.
Takeaways & Limitations
Long sequences cause perceptual and planning errors to compound, driving systemic failures and high incomplete ratios in core media editing tasks.
Abstract
from arXiv · showhide
While GUI agents have made significant progress in web navigation and basic operating system tasks, their capabilities in professional creative workflows remain largely underexplored. To bridge this gap, we introduce Cutverse, a benchmark designed to systematically evaluate autonomous GUI agents in realistic media post-production environments. We curate expert demonstrations across 7 professional applications (e.g., Premiere Pro, Photoshop), covering 186 complex, long-horizon tasks grounded in authentic editing workflows, involving dense multimodal interfaces and tightly coupled interaction sequences. To support scalable evaluation, we develop a lightweight parser that transforms raw screen recordings and low-level interaction logs into structured, compositional GUI action trajectories with precise grounding. Extensive evaluations reveal that existing agents achieve only 36.0\% task success on realistic media editing tasks, underscoring the challenges posed by complex, long-horizon media post-production workflows in our benchmark.While current models demonstrate promising spatial grounding, multimodal alignment, and coordinated action execution, they remain limited in long-horizon reliability and domain-specific planning.
1 Introduction
CutVerse addresses the underexplored challenge of evaluating GUI agents in dense, long-horizon professional media workflows. It introduces a benchmark and supporting infrastructure for realistic, fine-grained assessment, revealing substantial reliability gaps in current agents.
- Professional media post-production combines dense interfaces, tightly coupled operations, and long execution horizons requiring precise spatial, temporal, and multimodal coordination.
- Existing benchmarks struggle to provide reproducible evaluation in high-fidelity, resource-intensive media editing environments.
- Its infrastructure parses raw multimodal interaction logs into structured GUI trajectories and supports scalable execution in a Windows-based virtual environment.
- CutVerse introduces 186 complex, long-horizon tasks across 7 professional applications grounded in authentic post-production workflows.
- Extensive evaluations expose a performance gap, with current models struggling on spatially dense layouts and compositional GUI actions.
2 Related Work
Prior GUI and media benchmarks emphasize general interaction or output quality, leaving professional, trajectory-level post-production editing insufficiently evaluated. CutVerse fills this gap with a benchmark spanning professional software, complex workflows, and live executable environments.
- AIGC and code-driven systems generally target high-level semantic alignment, visual consistency, or simple video stitching rather than fine-grained professional editing.
- General-purpose GUI agents remain focused on web navigation and operating-system tasks, while media post-production is severely underexplored.
- Existing media creative benchmarks primarily assess perceptual quality and semantic fidelity of generated outputs, not the interaction trajectories producing them.
- CutVerse shifts evaluation from static output assessment to dynamic, trajectory-based verification of professional media manipulation.
- CutVerse is characterized by 186 professional desktop tasks with Pro media support, human-curated data, live environments, and end-to-end workflows.
3 The CutVerse Benchmark
CutVerse benchmarks realistic media post-production through authentic, long-horizon GUI workflows spanning dense multimodal interfaces, cross-application operations, and precise spatiotemporal coordination. Its infrastructure combines structured trajectory parsing, resettable virtual environments, milestone-based evaluation, and a dataset designed to expose sustained-planning and visual-grounding demands.
- Infrastructure: CutVerse parses synchronized screen recordings and low-level I/O logs into structured multimodal trajectories grounded in visual states and semantic context.The parser aligns continuous demonstrations with discrete action sequences for evaluation and capability decomposition.
- Benchmark scope: CutVerse organizes realistic creative workflows into a benchmark spanning diverse professional media-editing tasks and functional domains.The dataset contains 186 human-verified tasks across nine functional domains and covers the production pipeline from asset management to visual tuning.
- Complexity: 239 steps is the peak execution horizon, while trajectories average 18.73 steps and cross-application workflows average 21.20 steps versus 17.56 for isolated applications.Audio rhythm editing averages 26.00 steps and masking and tracking averages 25.40 steps.
- Visual complexity: 46.07% of operations target timelines and 25.32% target layer and track controls, shifting evaluation toward pixel-level audio-visual grounding.These interfaces require continuous spatial adjustment and coordination rather than simple point-and-click navigation.
- Online execution: Agents operate online in resettable Windows virtual machines, continuously perceiving dynamic canvases, timelines, and audio waveforms while issuing low-level actions.This setting tests visual disambiguation, temporal synchronization, and recovery from cascading errors during execution.
4 Baseline
The baseline evaluates proprietary and open-source vision-language models through executed GUI actions using task instructions, recent execution history, and current screenshots. Models perform strongly on procedural operations but degrade sharply on core editing, with local milestone competence failing to translate into end-to-end task success.
- Evaluation setup: Five vision-language models are evaluated under a unified online execution framework, combining proprietary APIs with locally deployed open-source systems.The evaluated models include Claude-Opus-4.6, Gemini-3-flash, Qwen3-32B, UI-TARS-1.5-7B, and EvoCUA-32B.
- Evaluation setup: At each step, agents receive the task instruction, five previous execution screenshots with descriptions and code, and the current keyframe before executing actions.The setting evaluates actual pyautogui execution rather than passive action prediction.
- Procedural operations: 1.000 task and milestone success rates are achieved by every model for generative workflows in the procedural setup and file-management category.Performance also remains competitive for export and delivery and launch and setup.
- Core media editing: 0.095 task success is recorded for UI-TARS in masking, matting, and tracking, illustrating severe degradation on precise core editing tasks.The passage reports a broad performance drop across models when tasks require precise content manipulation.
- End-to-end reliability: 0.929 milestone success versus 0.333 task success for Claude in audio and rhythm editing exposes a substantial local-to-global performance gap.Milestone-level performance consistently exceeds end-to-end task success, especially in complex editing categories.
5 Analysis
Current GUI agents show basic local competence but struggle to sustain precise, coordinated execution across complex media-editing workflows. Failures arise from dense interfaces, long horizons, weak global perception, fine-grained grounding, and limited compositional action coordination.
- Milestone-Task Consistency Gap: Milestone-level performance exceeds end-to-end success, showing local planning competence without reliable completion of global editing workflows.Agents handle Launch, Export, and Preview more reliably than Effects and Visual Tuning or Audio and Rhythm Editing.
- Media Applications Complexity: Professional editing software causes substantial performance degradation compared with structured generation tools because it combines dense layouts with sustained multimodal operations.The disparity exposes difficulties maintaining cross-modal alignment and audio-visual grounding under information overload.
- Long-Horizon Multimodal Task Difficulty: 73 seconds across 25 atomic steps for masking and tracking contrasts with 22 seconds and fewer than 6 steps for procedural previewing.Longer horizons require agents to maintain alignment across evolving layouts, audio signals, and editing intents, allowing small errors to accumulate.
- Missing Compositional Action Space: Balanced LeftClick, KeyPress, and Drag distributions coexist with consistently low success rates, indicating that coordination—not action availability—limits complex editing.Key–mouse combinations and temporally synchronized operations cannot be decomposed into independent atomic steps.
- Qualitative Evaluation: Agents frequently miss domain-specific tools, unlabelled toolbars, and subtle interface elements in condensed professional software.This component misrecognition limits access to advanced editing features.
- Qualitative Evaluation: Pixel-level grounding failures prevent delicate temporal trimming and precise spatial adjustment on timelines and canvases.Agents often miss intended targets when exact coordinate selection is required.
- Qualitative Evaluation: Localized observations prevent agents from verifying global workspace state, triggering erroneous operations based on incomplete context.Complex key–mouse coordination for global zooming further restricts acquisition of macro-level context.
- Qualitative Evaluation: When screenshots show no obvious alteration, agents repeat identical commands and can become trapped in infinite operational loops.Static visual feedback causes failures to register state transitions.
6 Conclusion
CutVerse benchmarks computer-use agents in real-world media post-production and reveals a gap between structured procedural competence and sustained professional editing demands. The findings support high-fidelity evaluation focused on spatial grounding, temporal coordination, and compositional control.
- 6 Conclusion: CutVerse provides a systematic benchmark and scalable infrastructure for evaluating computer-use agents in real-world media post-production workflows.It targets sustained execution of complex editing tasks requiring precise spatial grounding, temporal coordination, and compositional control.
- 6 Conclusion: Agents handle structured procedural operations but systematically fail during sustained execution of complex editing tasks.The conclusion emphasizes authentic, high-fidelity evaluation environments as necessary for measuring these capabilities.
A Details for Benchmark
CutVerse defines a visually grounded benchmark around low-level GUI interaction, expert-authored trajectories, and a multimodal POMDP formulation. Its action space excludes structured metadata and emphasizes precise mouse, keyboard, and viewport operations.
- Action Space: The benchmark uses a unified atomic action space limited to low-level mouse and keyboard operations to enforce pure visual grounding.The vocabulary includes click, dragTo, write, hotkey, and viewport navigation operations.
- Data Construction: Ten professional creators authored the foundational data for all 186 tasks using expertise in traditional post-production software and AIGC workflows.Their procedure included formal task definition and recording ground-truth execution videos.
- Problem Formulation: CutVerse formulates continuous GUI interaction as a multimodal POMDP whose observations contain raw high-resolution interface images and action history.The formalization isolates agents from accessibility trees and DOMs, requiring visual grounding through manual compositional operations.
A.3 Detailed Task Specifications
CutVerse decomposes authentic media-production workflows into 186 curated tasks and evaluates them through milestones, multimodal QA verification, and task-centric structure. This design measures intermediate state transitions rather than relying only on final outputs.
- Task Coverage: CutVerse comprises 186 curated tasks covering a taxonomy of fundamental editing typologies across diverse software environments.The tasks represent the full spectrum of multimodal workflows and real-world creative demands.
- Milestone Evaluation: Milestone-driven evaluation decomposes continuous long-horizon workflows into intermediate steps because binary final-output success is insufficient.The parser evaluates progress at granular milestones instead of treating each complex task as a single outcome.
- Multimodal QA Verification: Context-specific multimodal QA pairs verify each milestone by checking visual interface state transitions before and after designated actions.The mechanism targets precise spatial-temporal changes rather than superficial clicks.
- Task-Centric Decomposition: Task-centric decomposition maps complex creative objectives into discrete categorical tasks for evaluating sustained operational capability.This avoids assessing a long-horizon production objective as a monolithic black box.
A.4 Agent Implementation Details
CutVerse evaluates five multimodal GUI agents through unified screenshot-based execution pipelines, while accommodating model-specific prompts, action schemas, coordinate conventions, and interaction histories.
- Unified Agent Pipeline: Five multimodal foundation models are instantiated as autonomous agents within a shared screenshot-driven execution pipeline.Each agent receives task instructions, desktop screenshots, and, where applicable, recent interaction history before producing structured actions.
- Prompt and Output Schemas: The benchmark adopts OSWorld-style computer-use prompting for most agents, with canonical GUI actions and structured tool-call outputs.Claude uses JSON-only responses, Qwen uses JSON objects inside XML tool-call delimiters, and other agents adapt the paradigm to model-specific formats.
- Claude Opus 4.6: Claude responses contain observation, reasoning, action, coordinates, text or scroll fields, and a milestone-completion flag in a deterministic JSON schema.The schema omits fields unnecessary for the selected action and reserves completion for the full task.
- Coordinate Conventions: Agent coordinate systems vary across implementations, including fixed 1280 × 720 pixels, normalized 0–1000 grids, and processed-image coordinates.The framework rescales model outputs to actual screen resolution using linear interpolation or coordinate-conversion utilities.
- Model-Specific Interfaces: UITars uses Python-style function calls, while EvoCUA extends XML-based tool calling with stateful key_down and key_up actions for modifier-held editing.These model-specific interfaces support different action representations and fine-grained multimedia interactions.
B.1 Additional Data Statistics
CutVerse’s task and interaction distributions vary substantially across software environments, with dominant task types and secondary action modalities reflecting each tool’s workflow demands.
- Task Distribution Heterogeneity: 61.5% of After Effects tasks and 50.0% of DaVinci Resolve tasks are Effects and visual tuning.Export and delivery is a universally mandatory concluding milestone across nearly all platforms.
- Interaction Modality Complexity: 71.3% of Keling interactions are Left click operations, while Key press operations account for 31.8% in Premiere Pro and 31.2% in Photoshop.Secondary interactions expose greater mechanical complexity in professional editing software than foundational clicking alone suggests.
- Task-Driven Action Mapping: Preview, check, and validation tasks use Left click for 71.4% of actions, while Launch and setup tasks use it for 69.1%.Task type determines the interaction modality required for execution.
B.2 Additional Evaluation Analysis
Evaluation analysis shows substantial incompletion and a gap between agents’ perceived and verified task completion, underscoring reliability challenges in complex media workflows.
- Failure Analytics and Execution Consistency: 31.7% of Claude tasks and 32.8% of Gemini tasks remain incomplete, while Qwen and UI-TARS fail to complete 51.6% and 55.9%.These failure rates span proprietary, open-weight, and UI-centric agents.
- Failure Analytics and Execution Consistency: The macro-task consistency gap peaks at 0.148 for Qwen and 0.116 for Claude.The gap is defined as the difference between perceived completion rate and strictly verified execution accuracy.
- Evaluation Structure: CutVerse formalizes media post-production tasks into nine categories evaluated through granular multimodal question-answer pairs.The taxonomy verifies spatial-temporal state transitions across authentic task instructions.