Source-linked AI summary
JarvisHub: An Open Harness for Canvas-Native Multimodal Creative Agents
Yunlong Lin, Zixu Lin, Zhaohu Xing, Biqiang Li, Chenxin Li, Haonan Wang, Haitao Wu, Hengyu Liu, Jianghai Chen, Kaituo Feng, Kaixin Li, Shawn Chen, Shijue Huang, Sixiang Chen, Tsung-Yi Ho, Wenxuan Huang, Xiangyan Liu, Xiaomeng Hu, Xuanhua He, Yan Sun, Yunqing Zhao, Zhiqin Yang, Zehan Wang, Zhengyang Tang, Tianyu Pang, Xiangyu Yue
TL;DR
Long-horizon creative work requires agents to maintain evolving project context beyond isolated prompt-output interactions, but existing systems only partially support this state. JarvisHub uses an editable canvas as shared memory and action space, and qualitative demonstrations show more inspectable, reusable agentic creative workflows across representative tasks.
Problem
Existing creative-agent systems only partially represent evolving project state, including references, drafts, relationships, feedback, and intermediate work, limiting study of sustained multimodal creation.
Method
JarvisHub represents multimodal artifacts, versions, dependencies, edits, and feedback as addressable canvas nodes and links that agents inspect, modify, and connect through a protocol bridge.
Results
Qualitative demonstrations across narrative media, interactive web development, and presentation generation show more inspectable and reusable long-horizon creative workflows.
Takeaways & Limitations
An open canvas-native harness supports project-state benchmarks, process-level evaluation, and structured trajectory data for future creative-agent research.
Takeaways & Limitations
The experiments are qualitative demonstrations rather than a completed benchmark or leaderboard, and final artifact quality depends on external models and tools.
Abstract
from arXiv · showhide
Creative AI is moving from single-step asset generation toward long-horizon multimodal production. Although recent generative models can synthesize high-quality images, videos, audio clips, UI elements, storyboards, slides, and other creative assets, real-world creative work requires more than isolated prompt-output interactions. It involves references, drafts, alternatives, edits, failed attempts, version relations, tool actions, evaluation signals, and human feedback, which together form an evolving project state. Existing prompt-based, chat-based, and node-based generation systems only partially support this state, as they often discard intermediate context, rely on linear conversations, or require manually specified workflows. Recent commercial systems indicate a shift toward agent-assisted creative production, but their closed architectures make it difficult to study how agents represent context, choose tools, revise artifacts, recover from failures, and maintain consistency over time. To address this gap, we introduce JarvisHub, a canvas-native creative agent harness for long-horizon multimodal creation. JarvisHub treats an editable canvas as the user workspace, the agent's external memory, action space, and shared project state, representing multimodal artifacts, dependencies, versions, and feedback as typed canvas nodes and links. Through a three-layer architecture of canvas state, protocol bridge, and agent runtime, JarvisHub enables agents to act within an inspectable and editable creative state. This design moves creative agents beyond isolated tool use toward sustained, human-steerable creative automation, where agents can progressively plan, generate, revise, and organize multimodal projects while users remain able to inspect, guide, and intervene throughout the process.
1 Introduction
JarvisHub addresses the difficulty of sustaining multimodal creative work across evolving project state, references, alternatives, edits, feedback, and failures. It provides an open, canvas-native harness where agents can inspect, update, and organize this state throughout long-horizon creation.
- Motivation: Creative work extends beyond isolated prompt-output interactions, requiring agents to maintain references, spatial layouts, version branches, feedback, and unfinished artifacts.The central challenge is providing a workspace agents can read, update, and keep consistent across an extended workflow.
- JarvisHub: JarvisHub represents prompts, references, multimodal assets, candidate results, version relations, edits, and feedback as editable canvas nodes and links serving as shared project state.The canvas functions simultaneously as the user workspace, the agent’s readable and updateable state, and the structure for organizing creative materials.
- Limitations of Existing Systems: Existing prompt-to-output and chat-based systems only partially support long-horizon creation because they discard intermediate decisions or retain context mainly as a linear conversation.These limitations obscure failed trials, alternative candidates, revision history, and other evolving project state.
- Research Gap: Closed commercial creative products make it difficult to study project-state representation, action checking, tool scheduling, feedback use, and failure repair.Researchers can observe user-facing features but cannot inspect the underlying mechanisms that maintain context over long workflows.
- Contributions: JarvisHub combines canvas state, a protocol bridge, and an agent runtime to support controlled execution, synchronized updates, and trajectory recording.The harness formalizes long-horizon multimodal creation as an agent process over an editable project graph and demonstrates it across narrative media, interactive web development, and presentation decks.
2 Method
JarvisHub organizes long-horizon multimodal creation around an editable canvas that serves as shared project state, external memory, and action space. Its canvas state, protocol bridge, and agent runtime support inspectable artifact relations, constrained actions, recoverable updates, and feedback-driven iteration.
- Canvas state: JarvisHub represents creative projects as a typed canvas graph storing multimodal artifacts, dependencies, versions, editable contents, provenance, runtime status, and user interaction records.Relations encode reference use, version lineage, generation dependency, grouping, and workflow continuation.
- Canvas state: Canvas artifacts are addressable, reusable, and dependency-traceable, allowing agents and users to distinguish candidates, reuse intermediate results, and inspect how materials inform later outputs.This representation supports reference-guided generation, editing, rejection, and continued workflow development without relying on ambiguous conversational phrases.
- Protocol bridge: The protocol bridge exposes a capability manifest and execution grant, validates permitted mutations, and commits actions and evidence to the canvas so updates remain explicit, auditable, and recoverable.Committed updates can create nodes, modify fields, connect dependencies, attach artifacts, record failures, create checkpoints, or request human correction.
- Agent runtime: The agent runtime grounds each accepted action in the observed canvas, available capabilities, and current execution grant before invoking a capability and committing its result through the bridge.This contract prevents the runtime from maintaining a separate hidden project state.
- Agent runtime: Skills, memory, and subagents help sequence executable tool capabilities across long creative tasks while preserving state in the shared canvas, execution grants, and protocol-checked updates.Skills provide reusable procedures, memory preserves preferences and prior decisions, and subagents support branching subtasks whose results are integrated into the shared canvas graph.
- Feedback and trajectories: JarvisHub records feedback together with the state and execution context, enabling users, evaluators, or critics to guide repairs and follow-up decisions during creation.Feedback can select or reject candidates, revise prompts, adjust styles, identify defects, and evaluate consistency, evidence, visual quality, or task-specific constraints.
3 Experiments
JarvisHub is evaluated on three representative long-horizon creative tasks that require planning, persistent context, intermediate artifacts, feedback, and revision. Across paired workspace traces and final artifacts, the canvas keeps production processes inspectable while supporting coherent, consistent deliverables.
- Task coverage: The experiments cover narrative media generation, interactive web development, and presentation deck generation as representative long-horizon creative workflows.Each task develops an open-ended goal into a coherent deliverable through planning, reference organization, intermediate artifact generation, feedback, and local revision.
- Evaluation views: Each task pairs a canvas-centered workspace trace with a final generated artifact, exposing plans, references, assets, dependencies, progress, and accumulated project state.These paired views show how the production process remains inspectable and how canvas state supports coherent outputs.
- Narrative media generation: Narrative media generation turns a short-drama prompt into a coherent visual sequence while preserving story planning, visual references, dependencies, narrative continuity, and visual continuity across shots.The final artifact contains recurring characters, setting cues, and action continuity across shots.
- Interactive web development: Interactive web development turns an aesthetic and interaction brief into a rendered photography website while keeping design references, implementation progress, previews, revisions, and interface consistency inspectable.The task uses a light, Awwwards-inspired design with rich animations.
- Presentation deck generation: Presentation deck generation turns a machine-learning lecture topic into a structured slide deck while exposing content planning, visual assembly, previews, progress, and preserving topic structure and visual style.The task creates a decision-tree presentation styled like a Stanford lecture.
4 Discussion
JarvisHub treats the canvas as a shared, inspectable workspace and external memory where creative project state is represented through typed, addressable nodes and links. This open harness supports project-state benchmarks and trajectory-based analysis that evaluate creative-agent processes beyond final artifacts alone.
- Canvas as agent workspace: JarvisHub represents prompts, references, candidates, edits, versions, dependencies, and feedback as typed, addressable canvas nodes and links.The canvas thereby serves as both an external memory and an action space for the agent.
- Open research infrastructure: An open creative-agent harness can support project-state benchmarks specifying initial canvases, reference materials, tools, constraints, feedback events, and expected checkpoints.These benchmarks extend evaluation beyond a prompt and target answer.
- Trajectory-based analysis: Creative-agent analysis should examine trajectories because final artifacts alone do not fully characterize behavior when multiple outputs may be acceptable.Trajectories record how agents move between canvas states, including the context they use and actions they select.
5 Conclusion and Limitations
JarvisHub is a canvas-native agent harness for long-horizon multimodal creation that represents projects as editable canvas graphs and supports inspectable, reusable workflows. Its limitations include qualitative evaluation, dependence on external models and tools, and no guarantee that agent decisions are semantically correct.
- Conclusion: JarvisHub presents a canvas-native agent harness for long-horizon multimodal creation.It represents creative projects as editable canvas graphs.
- Conclusion: Agents can inspect context, call tools, update artifacts, incorporate feedback, and record state changes through a protocol bridge.The runtime combines canvas operations, media generation, native tools, repair, checkpointing, and trajectory capture in one shared workspace.
- Conclusion: Experiments on representative creative tasks show that JarvisHub supports more inspectable and reusable agentic creative workflows.The experiments are described as qualitative demonstrations rather than a completed benchmark or leaderboard.
- Limitations: Final artifact quality depends on the external models and tools used by the runtime because JarvisHub focuses on orchestration and project-state management.This limitation concerns the system’s scope rather than the canvas workflow itself.
- Limitations: Explicit and recoverable canvas actions do not guarantee that the agent’s creative decisions are semantically correct.The protocol bridge makes actions explicit and recoverable, but semantic correctness remains unguaranteed.
6 Contributions
The paper lists contributors alphabetically by first name, identifying three core contributors, twenty-one additional contributors, and two academic advisors.
- The contributors’ names are listed in alphabetical order from A to Z by first name.
- Core contributors are Yunlong Lin, Zixu Lin, and Zhaohu Xing.
- Additional contributors are Biqiang Li, Chenxin Li, Haonan Wang, Haitao Wu, Hengyu Liu, Jianghai Chen, Kaituo Feng, Kaixin Li, Shawn Chen, Shijue Huang, Sixiang Chen, Tsung-Yi Ho, Wenxuan Huang, Xiangyan Liu, Xiaomeng Hu, Xuanhua He, Yan Sun, Yunqing Zhao, Zhiqin Yang, Zehan Wang, and Zhengyang Tang.
- Academic advisors are Tianyu Pang and Xiangyu Yue.