PaperlayerView exact revision on arXiv

Source-linked AI summary

Omni-IO Skills: Harnessing Your Agent Omni-Native

Yanlin Li, Mingyang Hao, Shengqiong Wu, Hao Fei, Mong-Li Lee, Wynne Hsu

arXiv:2609.31847v1cs.CL

TL;DR

Multimodal production remains fragmented, and coordinating specialist tools leaves procedures, dependencies, intermediate assets, and revisions unresolved. Omni-IO Skills adds a plug-and-play harness with hierarchical Skills and persistent asset coordination around existing agents. On UniM-90, it broadens support and raises relative SQCS for two host agents.

  • Problem

    General-purpose agents face fragmented multimodal production capabilities, while specialist tools leave workflow procedures, dependencies, intermediate assets, and cross-turn revisions to be coordinated.

    Source: §1 · p. 2

  • Method

    Omni-IO Skills adds hierarchical Skills, a multimodal execution interface, dependency-aware orchestration, and a persistent Asset Registry around an unchanged host agent.

    Source: §1 · p. 2, §7 · p. 12

  • Results

    On UniM-90, input support reaches 100% for both hosts, while relative SQCS rises by 47.95 points for GPT-5.6 Sol and 49.96 points for Claude Sonnet 5.

    Source: §1 · p. 2, §7 · p. 12

  • Takeaways & Limitations

    Harness-level composition provides a practical route to Omni capability while keeping procedures, providers, and assets independently extensible.

    Source: §7 · p. 12

Abstract

from arXiv · show

General-purpose agents can plan, reason, and act over long horizons, yet their production capabilities remain fragmented across text, images, audio, video, documents, 3D assets, and code. Extending a foundation model to additional modalities ties capability growth to costly model updates, while assembling specialist models and tools leaves unresolved how procedures, dependencies, intermediate assets, and cross-turn revisions should be coordinated. We present Omni-IO Skills, a plug-and-play Agent Harness that makes existing agents omni-native through hierarchical Skills, a standardized multimodal execution interface, dependency-aware orchestration, and a persistent Asset Registry. Multi-asset workflows are represented as Declare Execution Graphs, which schedule independent operations concurrently and register successful outputs for downstream and cross-turn reuse across replaceable execution backends. Its 27 Skills cover 38 representative tasks spanning seven artifact modalities and four capability families: understanding, generation, reasoning, and retrieval. On UniM-90, the harness raises the input-support rates of GPT-5.6 Sol and Claude Sonnet 5 from 40.00% and 38.89% to 100%, while increasing relative Semantic--Quality Coupled Score from 26.99 to 74.94 and from 27.82 to 77.78, respectively; Strict Structure Score reaches 100.00 and 99.78. These results establish harness-level capability composition as a practical route to broad, evolvable Omni systems without changing the host agent's reasoning core.

1 Introduction

Multimodal production workflows need continuity across artifacts, while expanding foundation models and coordinating specialist tools leave distinct challenges. Omni-IO Skills adds a plug-and-play harness around an unchanged host agent, and on UniM-90 improves modality coverage and quality for two hosts.

1 Introduction

  • Multimodal production requires interleaved inputs and outputs while preserving semantic and asset continuity across workflows.The paper notes that artifacts can share facts, style, timing, and production constraints.Source: §1 · p. 2
  • Existing agents provide long-horizon planning, but multimodal production still requires external capabilities and explicit orchestration of dependencies, intermediate assets, failures, and prior outputs.The paper frames the gap as a harness between agent reasoning and heterogeneous execution backends.Source: §1 · p. 2
  • Omni-IO Skills is a plug-and-play harness that leaves the host agent unchanged and combines reusable Skills with execution and persistent asset layers.Its Atomic, Expert, and Scenario Skills cover 38 representative tasks spanning seven artifact modalities and four capability families.Source: §1 · pp. 2–3
  • 47.95 and 49.96 points: relative SQCS rises to 74.94 for GPT-5.6 Sol and 77.78 for Claude Sonnet 5, respectively.On UniM-90, input-support rates rise from 40.00% and 38.89% to 100%, and Strict Structure Scores reach 100.00 and 99.78.Source: §1 · pp. 2–3
  • The authors organize their contributions around a no-retraining harness, an extensible execution and asset architecture, broad task coverage, and gains across two host agents.The system covers seven artifact modalities, 27 Skills, and 38 representative tasks.Source: §1 · p. 3

2 Related Work

Omni foundation models unify modalities within the model, while agent harnesses and Skills coordinate capabilities at inference time. Omni-IO Skills brings these strands together for multi-asset workflows with persistent artifact state.

2.1 Omni Foundation Models

  • Unified autoregressive and hybrid discrete-continuous designs offer different trade-offs in interleaving, sequence length, perceptual detail, and coordination across specialized components.The paper positions Omni-IO Skills as complementary, unifying task execution and artifact flow across replaceable backends.Source: §2.1 · p. 3

2.2 Agent Harnesses and Skills

  • A harness adds operational support for sustained execution, while existing multimodal-agent work focuses largely on evidence acquisition, question answering, cross-modal reasoning, and response integration.Multi-artifact production additionally requires intermediate-asset transfer, provenance, and cross-turn revision.Source: §2.2 · p. 3
  • Skills encode when procedures apply, their required inputs and tool use, and expected outputs, preserving reusable workflows and composition patterns.Prior examples address expert tasks, computer use, and visual decision making.Source: §2.2 · p. 3
  • The intersection of multimodal understanding and generation with multi-asset execution and persistent state remains underdeveloped; Omni-IO Skills combines hierarchical procedures with traceable workflows and cross-turn asset reuse.Source: §2.2 · p. 4

3 System Capabilities and Task Coverage

Omni-IO Skills exposes an application-facing interface for requests that combine source material, intermediate assets, and deliverables across media types. Its task families support understanding, generation, reasoning, and retrieval, while higher-level Skills can compose tasks and share intermediate assets.

Modality Coverage

  • Figure 2 traces practical requests to modality flows, with icons to the left of arrows denoting inputs and icons to the right denoting outputs.Multiple icons on either side indicate tasks that consume or produce multiple artifact types.Source: §3 · p. 4

Cross-Modal Understanding

  • Understanding Skills turn heterogeneous source material into structured evidence for inspection and reuse, supporting both focused analysis and multi-source tasks.The resulting text can answer a request or provide requirements and references for later generation.Source: §3 · p. 4

Cross-Modal Generation

  • Generation Skills support direct transformations, single-asset creation, and coordinated production of multiple outputs with shared content and style.Examples include deriving promotional copy and imagery from a podcast recording or producing a concept image, showcase video, and 3D model from a game-character brief.Source: §3 · p. 4

Cross-Modal Reasoning

  • Reasoning Skills combine extracted evidence with user constraints to produce decisions, plans, or revised artifacts.A debugging task may return both an explanation and modified code.Source: §3 · p. 4

Cross-Modal Retrieval

  • Retrieval Skills obtain information from documents or the web and return it to the user or pass it to another Skill as grounded source material.Supported tasks include manual question answering, literature discovery, job-information search, and news retrieval.Source: §3 · p. 4

Task Composition

  • Higher-level Skills can select multiple task leaves, share intermediate assets, and assemble requested deliverables across application domains.Appendix Table 5 maps tasks to Skills.Source: §3 · p. 4

4 Layered Architecture of Omni-IO Skills

Omni-IO Skills separates task knowledge, tool interfaces, service implementations, and persistent outputs across four layers. Its hierarchical Skills expand user requests into executable workflows while keeping capabilities and artifacts reusable.

4.1 Architecture Overview

  • Four layers separate Skill-based task knowledge, standardized tool interfaces, provider bindings, and persistent assets between the host agent and multimodal tools.The Skill Entry expresses workflows as Declare Execution Graphs (DEGs), while lower layers realize their nodes.Source: §4.1 · pp. 4–6, Figure 3 · p. 6

4.2 Skill Entry Layer

  • Skills are organized by task granularity and compositional scope—not modality—as Atomic operations, Expert deliverable workflows, and Scenario application contexts.For example, a poster requires an Expert Skill, while coordinated social-media materials may require a Scenario Skill.Source: §4.2 · p. 6, Figure 4 · p. 7

4.2.1 Declarative Skill Representation and Hierarchical Expansion

  • Each Skill declares its applicability, required inputs, procedure, expected outputs, and relationships to other Skills, enabling selection and composition without service implementation details.The host agent selects Skills by task context and requested deliverables, then recursively expands higher-level Skills until executable tasks remain.Source: §4.2.1 · p. 6

4.2.2 Hierarchical Omni-IO Skills

  • Atomic Skills provide executable primitives; Expert Skills assemble and inspect one final deliverable, while Scenario Skills coordinate related deliverables and reuse assets across them.Expert workflows can revise only affected components while reusing validated intermediate assets; Scenario Skills clarify ambiguous task boundaries when needed.Source: §4.2.2 · pp. 6–7

4.3 MCP Tool Service Layer

  • The MCP Tool Service maps executable task specifications to understanding, generation, and utility tools through a common contract, normalizing external results for asset registration.Supported operations such as code and Markdown generation can instead run on the host agent under the same DEG dependency semantics.Source: §4.3 · p. 7

4.4 Provider and Configuration Layer

  • The Provider and Configuration layer binds semantic capabilities to concrete tools, providers, models, credentials, defaults, and fallback policies; implementations can change without rewriting Skills or DEGs.Node-level parameters can specialize an invocation without changing shared provider defaults.Source: §4.4 · p. 8

4.5 Asset Registry Layer

  • The Asset Registry records user-provided, intermediate, and final artifacts as path-independent assets with generation context and provenance, persisting versions for traceable downstream and cross-turn reuse.Its append-only JSON registry uses atomic, locked writes so parallel nodes or sessions cannot overwrite one another.Source: §4.5 · p. 8, Figure 5 · p. 9

5 Dependency-Aware Orchestration and Asset Lifecycle

Omni-IO Skills coordinates multimodal work as dependency-aware execution graphs and preserves successful artifacts for downstream and cross-turn reuse. Its runtime validates and schedules tasks, routes them to appropriate backends, and tracks their outputs.

5.1 Runtime Workflow

  • Skill selection and graph construction organize control flow, while registered artifacts from user, historical, or generated sources supply inputs to downstream tasks.Tasks are validated and scheduled in Waves; Figure 5 summarizes the interaction between control flow and asset flow.Source: §5.1 · p. 9

5.2 DEG Representation and Construction

  • A DEG represents executable tasks and existing-asset sources as nodes, with shared intermediate results referenced once and dependencies recorded explicitly.Historical assets become completed source nodes after their references resolve, without triggering generation.Source: §5.2 · p. 9

5.3 Graph Validation, Wave Scheduling, and Failure Handling

  • Validation rejects unresolved references and cycles; each Wave runs mutually independent ready nodes concurrently, while failed nodes cancel pending descendants but leave independent branches running.Completed outputs are not rolled back when another node fails.Source: §5.3 · p. 10

5.4 Runtime Tool Routing and Parameter Injection

  • Task type selects a tool and provider configuration; predecessor outputs are injected according to the dependency, while native host tasks follow the same input and output contract.Effective parameters combine provider defaults, node overrides, and inputs from predecessor outputs.Source: §5.4 · p. 10

5.5 Asset Propagation and Cross-Turn Reuse

  • Successful outputs enter the Asset Registry for downstream use and cross-turn reuse; revisions create new records, and registry writes are atomic.In the product-promotion example, four Waves produce deliverables, and a later landing-page revision reruns only that task using registered assets.Source: §5.5 · p. 10, Table 2 · p. 11

6 Experiments

The experiments compare each host agent with and without Omni-IO Skills on UniM-90 and examine both benchmark metrics and qualitative workflows. Both hosts improve across input support, semantic quality, coherence, and output structure, and the case studies illustrate coordinated multimodal production.

6.1 Experimental Setup

  • The setup tests two hosts on a fixed 90-instance UniM subset spanning seven modalities and interleaved combinations, using within-agent comparisons.The evaluation reports input support and absolute and relative variants of SQCS, ICS, StS, and LeS.Source: §6.1 · p. 11

6.2 Main Results

  • On UniM-90, relative SQCS rises from 26.99 to 74.94 for GPT-5.6 Sol and 27.82 to 77.78 for Claude Sonnet 5; relative ICS rises from 34.61 to 93.98 and 32.00 to 83.28.Input-support rates increase from 40.00% and 38.89% to 100% for the respective hosts.Source: §6.2 · pp. 11–12

6.3 Case Studies

  • In two case studies, both hosts produce requested art-tutorial and product-promotion deliverables while preserving consistency across tutorial steps, product identity, and outputs.The workflows combine video and audio understanding with tutorial-image generation, and product understanding with poster, video, and landing-page creation.Source: Figure 6 · p. 12, §6.3 · p. 12

6.2 Main Results

  • Absolute SQCS increases from 67.49 to 74.94 for GPT-5.6 Sol and 71.53 to 77.78 for Claude Sonnet 5; StS reaches 100% and 99.78%, respectively.Both hosts attain 100% LeS.Source: §6.2 · p. 12

7 Conclusion

Omni-IO Skills composes multimodal capabilities around existing agents without altering their reasoning core. The conclusion presents harness-level composition as a practical route to extensible Omni systems.

7 Conclusion

  • The harness combines hierarchical Skills, dependency-aware DEGs, replaceable specialist backends, and a persistent Asset Registry without changing the host agent’s reasoning core.On UniM-90, both hosts reach 100% input support, with relative SQCS gains of 47.95 and 49.96 points.Source: §7 · p. 12

A Complete Catalog of Omni-IO Skills

  • The catalog covers Atomic, Expert, and Scenario Skills.It is presented as a complete catalog of Omni-IO Skills.Source: Table 4 · p. 16

B Representative Real-World Task Coverage

Omni-IO Skills covers representative multimodal tasks through configurable tools and hierarchical Skills, then plans, coordinates, assembles, and revises related deliverables while reusing assets.

B Representative Real-World Task Coverage

  • The task coverage spans representative real-world configurations, while tools expose multimodal operations and assembly and inspection support completed, locally revised artifacts.Controllable assembly can typeset exact poster copy over a generated background; defect attribution supports minimal rework while reusing validated assets.Source: Table 5 · p. 17, §C.1 · p. 17, Table 6 · p. 18

C.2 Provider Bindings and Fallback

  • Provider bindings and node parameters are configurable without changing Skill specifications, while native or local capabilities remain available without a complete credential set.Effective generation parameters are recorded in asset records; configuration checking reports ready and missing capabilities and free alternatives when available.Source: §C.2 · p. 18, §C.3 · p. 18, §C.4 · p. 18

D.1 Shared Specification Schema

  • Skill specifications declare capabilities at three levels, are loaded on demand, and use downward-only invocation to keep workflow expansion finite and the resulting DEG acyclic.Atomic Skills declare tools, Expert Skills declare workflows, and Scenario Skills declare deliverable-selection rules; specifications do not bind services, models, or API keys.Source: §D.1 · p. 19, §D.2 · p. 19, Table 9 · p. 20

D.3 Skills Extension

  • Scenario Skills select lower-level capabilities for an application context, Expert Skills define deliverable workflows, and Atomic Skills add executable operations.Scenario Skills declare deliverable selection and cross-deliverable relationships; Expert Skills decompose production into existing Atomic Skills and define assembly, inspection, and localized revision.Source: §D.3 · p. 20

E.1 Input and Expansion Contracts

  • Expert Skills translate deliverable goals and constraints into dependent executable tasks, assigning generative content to Skills and precision-sensitive work to controllable processing or assembly.They emit flat task fragments for validation and scheduling by the common execution procedure, with modality-appropriate inspection.Source: §E.1 · pp. 20–21, Table 10 · p. 21

F.1 Input and Planning Contracts

  • Scenario Skills preserve explicit user scope, select deliverables, and assign each to the lowest Skill level capable of its required procedure.The plan can share an identity anchor, run dependent outputs in parallel, reuse existing assets, and aggregate deliverables.Source: §F.1 · p. 23, Table 12 · p. 23

F.2 Cross-Deliverable Coordination and Reuse

  • A common scenario brief propagates locked facts and shared direction, while explicit artifact dependencies preserve asset identity and permit independent branches to run in parallel.Outputs that share only the brief use self-contained prompts and need no shared artifact dependency.Source: §F.2 · p. 23

F.3 Completion and Revision

  • Scenario-level completion checks deliverable states and relationships, while scoped revisions update affected briefs and dependencies and retain completed assets as new versions are registered.If a shared dependency fails, dependent outputs become unavailable while independent branches may complete.Source: §F.3 · pp. 23–24

G.1 Reference Orchestration Procedure

The reference procedure expands Skills into terminal tasks, validates and schedules their dependency graph, and uses registry state to reuse completed assets and handle failures.

G.1 Reference Orchestration Procedure

  • The orchestration procedure validates a DEG, schedules dependency-ready nodes in parallel Waves, registers successful outputs, and propagates failures while independent branches continue.Node status controls scheduling and cancellation, while the registry records successfully materialized artifacts for later use.Source: §G.1 · pp. 24–25

G.2 Node Schema

  • Admissible node types map mainly to fixed-signature tool calls, while native code and markdown nodes can consume multiple upstream assets and require explicit registration.Source: Table 14 · p. 25
  • A DEG node is either a generation node with a self-contained prompt or an asset-source node with an asset_ref, never both.Prompts do not refer to another node's prompt; semantic consistency across concurrent nodes comes from their shared planning context.Source: §G.2 · p. 25, Table 13 · p. 25

G.3 Task Types and Execution Sites

Execution rules validate task graphs before dispatch, preserve independent work through failures and interruptions, and maintain registered assets for versioning and cross-turn reuse.

G.3 Task Types and Execution Sites

  • Only natively executed code and markdown tasks can consume multiple upstream assets; most tool-call task types have fixed parameter signatures.Source: §G.3 · p. 25

G.4 Validation Rules

  • Every DEG validation rule is decidable from the declaration, and a violation returns the graph to planning rather than execution.When dependency necessity is unclear, the system declares the dependency.Source: §G.4 · p. 25, Table 15 · p. 26

G.5 Failure and Interruption Semantics

  • Failures and user interruptions cancel affected downstream work while independent branches continue, and completed assets remain registered.Pending or in-flight affected nodes may be cancelled, edited, or re-issued according to the revision scope.Source: §G.5 · p. 25, Table 16 · p. 26

G.6 Worked Example

  • In the worked graph, a reused asset is complete on entry, two independent generations run concurrently, and a failed generation would cancel only the dependent aggregation.The other generation can still complete if the 3D node fails.Source: §G.6 · p. 26, Table 17 · p. 26

H.1 Record Schema

  • Registry records include descriptions for semantic retrieval and turn identifiers to distinguish same-type assets produced in different turns.Records are keyed by asset_id and persisted in a workspace JSON file.Source: §H.1 · p. 26, Table 18 · p. 27

H.2 Registration Paths

  • MCP generation tools register outputs before returning, whereas host-written code, markdown, and assembled files require an explicit registration call.Source: §H.2 · p. 27

H.3 Concurrency and Versioning

  • The workspace registry serializes concurrent writes, and append-only records give regenerated artifacts distinct, individually retrievable versions.A new record may point back to the record from which the regenerated artifact was derived.Source: §H.3 · p. 27, Figure 11 · p. 28

H.4 Cross-Turn Reuse

  • Cross-turn reuse retrieves an existing asset by semantic match and inserts it as a completed asset-source node, avoiding a new generation call.The figure depicts this reuse together with parallel generation and downstream aggregation.Source: Figure 10 · p. 27, §H.4 · p. 28

I Experiment Details

The experiment uses a fixed, difficulty-stratified 90-instance subset and compares base agents with configurations augmented by Omni-IO Skills under controlled settings.

I.1 Construction of UniM-90

  • UniM-90 contains 90 cases sampled from 30 UniM domains, with one easy, medium, and hard case per domain when available.An unavailable difficulty level is replaced by another available level from the same domain.Source: §I.1 · p. 28

I.2 Configuration

  • The comparison adds Omni-IO Skills and its MCP services while keeping other settings and provider bindings fixed, isolating the harness contribution.Source: §I.2 · p. 28
Loading 2609.31847v1…