PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniHarness: Harnessing Generalizable Visual Generation via Symbolic Policy Learning

Xu Xu, Jinxiu Liu, Zhangbo Qiao, Jiaxing Lu, Xiangyu Zhang, Yubin Gu, Fangwei Ning, Yan Shi

arXiv:2609.16057v1cs.LGcs.AI

TL;DR

Existing visual-generation methods can retain task-specific solutions, defer reflection until task completion, or learn only when downstream tasks demand it. OmniHarness learns reusable symbolic policies through self-directed practice and feedback-guided execution. It reports 95.0% resolve on ComfyBench’s Creative tasks, 27.5 percentage points above the strongest baseline.

  • Problem

    Existing methods often distill task-specific experience without revealing task-family principles, reflect only after completion, or acquire knowledge only in response to downstream demands.

    Source: §1 · pp. 2–3

  • Method

    OmniHarness abstracts verified executions into reusable symbolic policies, adapts and composes them for new tasks, verifies intermediate outputs, and learns through practice tasks before downstream objectives are specified.

    Source: §1 · p. 3

  • Results

    95.0% resolve rate on ComfyBench’s Creative tasks exceeds the strongest baseline by 27.5 percentage points, and experiments across six benchmarks demonstrate effectiveness.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    Frozen policy snapshots support plug-and-play reuse across visual agent frameworks, while execution feedback refines policies without model fine-tuning.

    Source: §1 · p. 3, §5 · p. 10

    Self-directed inquiry and downstream execution add computation, agent-side token consumption, and reasoning time, especially for complex tasks.

    Source: §A.9 · p. 38

Abstract

from arXiv · show

Unified multimodal large language models (MLLMs) and multi-agent systems have advanced visual generation. However, three limitations remain. (1) Existing methods often distill task-specific experience with limited generalizability. (2) Reflection is often deferred until task completion. (3) Knowledge is often acquired only in response to downstream task demands. To address these limitations, we introduce OmniHarness, a framework for generalizable visual generation via symbolic policy learning. OmniHarness abstracts verified executions into symbolic policies for visual generation task families, capturing shared procedures and applicability conditions while removing instance-specific inputs. The harness instantiates, adapts, and composes these policies for new tasks. Intermediate verification guides refinement and failure recovery during execution. Through self-directed inquiry, OmniHarness autonomously generates and executes practice tasks near its capability limits before downstream objectives are specified. Execution feedback continually refines the policies while model parameters remain fixed. Experiments across six benchmarks, three MLLM backbones, and three visual agent frameworks demonstrate strong performance and continual capability expansion. On ComfyBench's Creative tasks, OmniHarness achieves a 95.0% resolve rate, exceeding the strongest baseline by 27.5 percentage points. Frozen policy snapshots improve existing visual agent systems through plug-and-play reuse.

1 Introduction

OmniHarness addresses limits in generalization, timely error correction, and learning before downstream demands by learning reusable symbolic policies and refining them through execution. It combines self-directed inquiry with feedback-guided execution and reports a 95.0% resolve rate on ComfyBench’s Creative tasks.

Motivation

  • Existing methods often preserve individual solutions without revealing task-family principles, while reflection may wait until completion and learning may begin only after downstream demands arise.These gaps motivate a system that generalizes beyond individual cases, reflects as it acts, and learns through self-directed exploration.Source: §1 · pp. 2–3

Symbolic Policy Learning

  • OmniHarness abstracts verified executions into symbolic policies that capture shared procedures and applicability conditions while removing instance-specific inputs.The harness instantiates, adapts, and composes these policies for new tasks.Source: §1 · p. 3

Feedback-Guided Execution

  • Intermediate-output verification guides workflow refinement and failure recovery during execution, limiting error propagation.Figure 2 contrasts this approach with unified MLLMs lacking deliberate reasoning and self-correction, and multi-agent systems lacking unified coordination and persistent knowledge accumulation.Source: §1 · p. 3, Figure 2 · p. 3

Self-Directed Inquiry

  • Before downstream objectives are specified, OmniHarness generates and executes practice tasks near its capability limits, using execution feedback to guide policy learning.The framework continually refines policies while model parameters remain fixed; frozen snapshots support reuse across visual agent frameworks.Source: §1 · p. 3, Figure 3 · p. 4

Experimental Evaluation

  • 95.0% resolve rate on ComfyBench’s Creative tasks exceeds the state-of-the-art baseline by 27.5 percentage points.Experiments across six benchmarks demonstrate strong performance, continual capability expansion, and transfer across visual agent frameworks.Source: §1 · p. 3

2 Related Work

Related work develops unified multimodal models and multi-agent visual systems, but both leave gaps in verification, recovery, or reusable coordination. OmniHarness instead learns policies shared across visual-generation task families and uses them during execution.

End-to-End Unified MLLMs

  • Unified MLLMs combine visual understanding and generation, but standard inference offers limited explicit verification, failure recovery, and persistent workflow reuse.The literature also explores cross-modal chain-of-thought, multi-representation reinforcement, and shared-context visual tokenization.Source: §2 · p. 4

Agentic Systems

  • Visual-agent systems combine planning and feedback, while OmniHarness distinguishes itself by learning symbolic policies from verified executions for task-family reuse.OmniHarness adapts and composes those policies for new tasks and uses intermediate verification to guide refinement and recovery.Source: §2 · p. 4
  • Existing agentic approaches include evolving execution checks, synthesizing task-specific harnesses, and learning symbolic concepts from incoming tasks; OmniHarness also learns through practice tasks before downstream objectives are specified.Its execution feedback continually refines policies while model parameters remain fixed, and frozen snapshots support plug-and-play reuse across visual agent frameworks.Source: §2 · p. 4

3 OmniHarness: Symbolic Policy Learning

OmniHarness learns reusable visual-generation policies through self-directed inquiry, applies and verifies them during task execution, and updates its policy and failure libraries from outcomes.

3.1 Self-Directed Inquiry

  • The competence frontier score favors tasks with estimated competence near 0.5, while capability novelty favors underexplored context–capability pairs.Candidate tasks combine novelty and learnability scores; task generation filters near-duplicates and avoids known failure patterns.Source: §3.1 · p. 5

3.2 Feedback-Guided Execution

  • For practice or downstream tasks, the planner instantiates, adapts, and composes policies, then verifies workflow steps and repairs failed components while preserving verified steps.Execution returns an executable workflow and records status, verifier feedback, and evidence; verification can repeat until success or the retry budget is exhausted.Source: §3.2 · pp. 5–6

3.3 Symbolic Policy Learning

  • After verified success, OmniHarness distills the workflow into a task-family policy that captures shared procedures and applicability conditions while removing instance-specific inputs.Failures and verifier feedback are recorded for root-cause analysis and corrective strategies; inquiry-learned policies can be exported as frozen snapshots for external visual agents.Source: §3.3 · p. 6

4 Experiments

Across workflow construction, text-to-image generation, and image editing, OmniHarness performs strongly on benchmark tasks; ablations associate inquiry and feedback-guided execution with improved resolve rates.

4.1 Autonomous Workflow Construction

  • Codex GPT-4o + OmniHarness reaches 92.5% Total Resolve and 95.0% Creative Resolve, versus SymbOmni’s 67.5% on Creative tasks.On Complex tasks, OmniHarness matches SymbOmni at 83.3% Resolve, while ComfyMind reaches 85.0%.Source: §4.1 · pp. 6–7

4.2 Text-to-Image Generation

  • OmniHarness leads GenEval overall (0.997), GenEval2 on four skills, and WISE overall (WiScore 0.86, +0.06 vs SymbOmni and GPT-Image-1).Its GenEval2 Object score of 95.0 matches SymbOmni but remains below Qwen-Image and Gemini 2.5 Flash Image.Source: Table 2 · p. 7, §4.2 · p. 8, Table 3 · p. 8

4.3 Image Editing

  • In Understanding Scenarios, OmniHarness leads all four metrics (23.89 dB PSNR, 0.86 SSIM, 0.05 LPIPS, 24.55 CLIP Score); in Reasoning, it leads SSIM (0.80) and CLIP Score (21.32).Its Reasoning Scenarios LPIPS of 0.05 matches the best baselines at two-decimal precision.Source: §4.3 · p. 9, Table 5 · p. 10

4.4 Ablation Study

  • Removing inquiry reduces Creative Resolve from 95.0% to 72.5%; removing planning, verification, or localized recovery reduces Complex Resolve from 83.3% to 55.0%, 68.3%, or 76.7%, respectively.Removing inquiry also lowers Total Resolve from 92.5% to 87.0%; removing novelty or the competence frontier score lowers Creative Resolve to 87.5% or 85.0%; removing verification lowers Complex Pass from 100.0% to 75.0%.Source: §4.4 · p. 9, Table 6 · p. 10, Table 7 · p. 10

5 Conclusion

OmniHarness learns reusable symbolic policies for visual generation and applies them to new tasks with intermediate verification, self-directed inquiry, and execution feedback. The paper reports strong benchmark performance and plug-and-play reuse of frozen policies.

5 Conclusion

  • OmniHarness abstracts verified executions into reusable symbolic policies for visual generation task families.The policies capture shared procedures and applicability conditions while removing instance-specific inputs.Source: §5 · p. 10
  • The harness adapts and composes policies for new tasks, using intermediate verification to guide refinement and localized recovery.Source: §5 · p. 10
  • Self-directed inquiry acquires policies before downstream objectives are specified, while execution feedback refines them without model fine-tuning.Across six benchmarks, OmniHarness achieves 95.0% Resolve on ComfyBench Creative tasks, 27.5 percentage points above the strongest baseline; frozen snapshots improve external agents through plug-and-play reuse.Source: §5 · p. 10

A.1.1 Problem Formulation and Policy Learning Loop

OmniHarness formulates visual workflow construction as an agent that maps task instructions, scene context, and a ComfyUI knowledge graph to an executable workflow. Its inquiry loop selects practice tasks, runs them with feedback, and updates policy and failure libraries.

A.1.1 Problem Formulation and Policy Learning Loop

  • A visual workflow agent receives an external task instruction, scene context, and ComfyUI knowledge graph, and produces an executable workflow.Source: §A.1.1 · p. 18
  • Self-directed inquiry generates and executes practice tasks before downstream objectives are specified.The workflow library stores reusable symbolic policies with reliability evidence, while the failure library stores failure evidence and corrective strategies.Source: §A.1.1 · p. 18
  • Task selection balances capability novelty with estimated learnability, favoring underexplored context–capability pairs and tasks near the estimated competence frontier.Source: §A.1.1 · p. 18
  • During execution, Run plans, synthesizes a Code-as-Policy workflow, executes it, verifies intermediate outputs, and performs localized recovery.It uses the ComfyUI knowledge graph and instantiates, adapts, and composes policies from the workflow library.Source: §A.1.1 · p. 18
  • Verified executions update the policy library with shared procedures and applicability conditions, while failure evidence and remedies are recorded separately.Updates apply to practice and downstream tasks with model parameters fixed; the exported inquiry snapshot remains frozen for plug-and-play reuse.Source: §A.1.1 · p. 18

A.1.2 Exploration within Reach

OmniHarness selects practice tasks that combine underexplored capabilities with a learnable level of estimated competence. Task competence is conservatively estimated from applicable workflows and the weakest required capability.

A.1.2 Exploration within Reach

  • Candidate tasks specify a description, generation modality, required capabilities, and an optional source image.Text-to-image tasks use no source image.Source: §A.1.2 · p. 19
  • Capability novelty is higher for underexplored context–capability pairs, while averaging prevents larger capability sets from scoring higher solely because of their size.Source: §A.1.2 · p. 19
  • For each required capability, support is based on the most reliable applicable workflow, with ε = 0.05 used when none exists.Reliability discounts success rates based on limited execution evidence.Source: §A.1.2 · p. 19
  • Estimated task competence follows the weakest required capability.Source: §A.1.2 · p. 19
  • The competence-frontier score peaks at estimated competence 0.5 and downweights tasks with very low or very high estimated competence.Combined with capability novelty, it favors novel tasks near the competence frontier.Source: §A.1.2 · p. 19

A.1.3 Feedback-Guided Execution

Feedback-guided execution turns a task into a verified plan and executable workflow, checking outputs during execution. On failure, the harness diagnoses and repairs affected components while preserving verified steps.

A.1.3 Feedback-Guided Execution

  • Agents share the task and context together with plans, programs, verification results, and corrections.Source: §A.1.3 · p. 19
  • The planner decomposes each task into an ordered plan and uses the policy library, failure patterns, and remedies to construct it.A plan verifier checks step order, dependencies, and task alignment.Source: §A.1.3 · pp. 19–20
  • The workflow writer converts the verified plan into a Python-like Code-as-Policy program, where function calls represent ComfyUI nodes and data flow defines their connections.A reversible interpreter compiles the program into an executable workflow.Source: §A.1.3 · p. 20
  • The resulting execution instance preserves workflow node types, parameters, dependencies, and order, and records intermediate outputs.Source: §A.1.3 · p. 20
  • Verification checks plans, workflow validity, intermediate effects, and final-task alignment; failures trigger diagnosis and localized repair, followed by re-verification until success or the retry budget is exhausted.If a failure persists, a subagent can solve the affected step and return a reusable subworkflow.Source: §A.1.3 · p. 20

A.1.4 Symbolic Policy Learning and Export

Verified successful executions become reusable symbolic policies, while failures are recorded as corrective strategies. OmniHarness curates these libraries and exports a frozen snapshot for external visual agents.

A.1.4 Symbolic Policy Learning and Export

  • On verified success, an execution is abstracted into a policy entry that retains shared workflow structure and applicability conditions while removing task-specific inputs.Entries support instantiation, adaptation, and composition within a visual generation task family.Source: §A.1.4 · p. 20
  • Equivalent workflows are updated rather than duplicated, and usage and verified-success counts inform workflow tiers and retrieval.New entries begin Provisional, become Validated at uw ≥ 3 and pw ≥ 0.5, and become Suspended at uw ≥ 10 and pw ≤ 0.2.Source: §A.1.4 · p. 20
  • Failures are stored with task and execution evidence, and analysis derives corrective strategies with specified causes, remedies, and applicable scope.Source: §A.1.4 · p. 20
  • A curator periodically merges redundant workflows, refines their conditions and expected effects, consolidates corrective strategies, and updates statistics and reliability tiers.Repeated failures indicating a missing capability may trigger workflow construction and verification.Source: §A.1.4 · p. 20
  • After inquiry, OmniHarness exports a frozen snapshot containing both workflow and failure libraries for plug-and-play reuse by external visual agents.The snapshot receives no downstream updates, although the internal policy library continues learning from downstream feedback.Source: §A.1.4 · pp. 20–21

A.1.5 Overall Symbolic Policy Learning Procedure

This section presents OmniHarness’s procedure for learning reusable symbolic visual-generation policies through self-directed, verified practice and adapting and refining them during downstream execution, while exporting frozen snapshots for reuse. It evaluates the framework across six visual-generation benchmarks and examines frozen-policy reuse in host agents and video tasks, user-study ratings, computational costs, and implementation.

A.2 Self-Directed Inquiry Configuration

  • Inquiry typically uses 50 iterations, proposes ten practice tasks per iteration, and withholds downstream instructions, targets, workflows, annotations, and evaluation labels.For GenEval, GenEval2, and WISE, inquiry uses T2I capabilities without access to benchmark evaluation tasks. Experiments use ComfyUI; comparisons fix retries at four and match underlying generative models within each setting.Source: §A.2 · p. 21, §A.3 · p. 22

A.4.1 Fine-Grained Ablation of Self-Directed Inquiry

  • After 50 inquiry iterations, the workflow library contains 43 policies, led by Localized Image Editing at 16.3%.Verified workflows are abstracted by removing task-specific inputs; equivalent entries receive updated usage and success evidence, while new entries begin as Provisional. The library combines reusable workflows with failure evidence and corrective strategies.Source: §A.2 · p. 21, Figure 7 · p. 23, §A.4.1 · pp. 24–25
  • Creative Total Resolve falls from 95.0% to 72.5% without self-directed inquiry; 50 inquiry iterations yield 92.5% overall Resolve.At 100 iterations, Creative Resolve reaches 97.5% and overall Resolve 93.5%, only 1.0 percentage point above 50 iterations. Removing capability novelty or the competence frontier score lowers Creative Total Resolve to 87.5% or 85.0%, respectively.Source: Figure 8 · p. 24, §A.4.1 · pp. 25–26

A.4.3 Symbolic Policy Learning Analysis

  • Downstream execution expands the library from 43 to 81 entries, including 23 video workflows, while direct and compositional reuse together account for 91.9% of 185 resolved tasks.The Validated share rises from 32.6% to 55.6%; direct reuse covers 90.7% of resolved Vanilla tasks, while compositional reuse covers 78.0% of resolved Complex tasks.Source: Figure 9 · p. 26, Figure 10 · p. 26, §A.4.3 · pp. 27–28

A.4.2 Fine-Grained Ablation of Feedback-Guided Execution

  • On Complex tasks, removing planning lowers Total Resolve from 83.3% to 55.0%, while removing intermediate verification lowers Total Pass from 100.0% to 75.0%.Removing localized recovery reduces Pass to 93.3% and Resolve to 76.7%.Source: §A.4.2 · p. 26

A.4.4 Reasoning Efficiency Analysis

  • Across 200 ComfyBench tasks, reasoning averages 12.03K tokens and 52.9 s per task, reaching 18.63K tokens and 83.5 s in the final Creative group.Creative tasks include longer instructions and complex text, layout, identity-preservation, and video requirements.Source: Figure 12 · p. 28, §A.4.4 · pp. 28–29

A.4.5 Backbone Ablation

  • 95.0% Total Resolve is the highest overall result among tested OmniHarness configurations, achieved by Codex GPT-5.5.All tested configurations achieve 100.0% Pass; GPT-5.5 and Codex GPT-4o each reach 92.5% Total Resolve.Source: Table 13 · p. 29, §A.4.5 · p. 29

A.5.1 Evaluation on ComfyBench

  • On ComfyBench, both OmniHarness variants reach a 100.0% Pass rate; Codex GPT-4o + OmniHarness achieves 92.5% Total Resolve and 95.0% Creative Resolve, beating SymbOmni by 6.5 and 27.5 points.The Creative Resolve rate also exceeds ComfyMind's 57.5% by 37.5 percentage points.Source: §A.5.1 · p. 31

A.5.2 Evaluation on GenEval

  • OmniHarness scores 0.997 overall on GenEval, exceeding SymbOmni, ComfyMind, and GPT-Image-1 by 0.017, 0.097, and 0.157, respectively.It scores 1.00 on single-object generation, co-occurrence, counting, color, and relative position, and 0.98 on attribute binding.Source: §A.5.2 · p. 31, Figure 14 · p. 32

A.5.3 Evaluation on GenEval2

  • On GenEval2, OmniHarness leads four of five skills, scoring 94.0 on Attribute and Count, 76.9 on Position, and 89.0 on Verb; its Object score of 95.0 matches SymbOmni.It remains below Qwen-Image and Gemini 2.5 Flash Image on Object, while its strongest advantages are on tasks with interacting compositional constraints.Source: §A.5.3 · p. 31, Figure 15 · p. 33

A.5.4 Evaluation on Reason-Edit

  • On Reason-Edit Understanding Scenarios, OmniHarness leads all four metrics; in Reasoning Scenarios, it has the highest SSIM (0.796) and CLIP Score (21.318), but trails on PSNR and LPIPS.Its GPT-score comparison ranks it first on Left-Right, Mirror, Color, Multiple-Objects, and Reasoning, and second on Relative-Size.Source: §A.5.4 · pp. 31–32

A.5.5 Evaluation on WISE; A.5.6 Evaluation on KRIS-Bench

  • OmniHarness reaches 0.86 overall WiScore, while its KRIS-Bench score is 77.33—5.15 points above SymbOmni and 23.97 above BAGEL-Think.On WISE it leads in Time, Biology, Physics, and Chemistry; on KRIS-Bench it leads in Conceptual performance but trails GPT-Image-1 overall by 2.76 points.Source: §A.5.5 · p. 32, §A.5.6 · p. 33, Table 15 · p. 36

A.6 Detailed Plug-and-Play Analysis

  • The same frozen Kinquiry snapshot improves Creative Resolve without model fine-tuning: ComfyAgent rises from 15.0% to 42.5% (+27.5 points), ComfyMind from 57.5% to 75.0% (+17.5), and SymbOmni from 67.5% to 77.5% (+10.0).Source: §A.6 · pp. 34–35, Table 16 · p. 36

A.7 Cross-Modal Generalization Evaluation

  • With its image-only frozen snapshot, OmniHarness achieves 85.9% overall Resolve on video tasks, 5.1 points above SymbOmni, and reuses image workflows alongside video-specific temporal components.Resolve rates are 84.2% on T2V, 92.0% on I2V, and 80.0% on V2V; the snapshot is unchanged during evaluation.Source: §A.7 · pp. 35–36, Table 17 · p. 38

A.8 User Study; A.9 Limitations; A.10 System Prompts

  • OmniHarness obtains the highest average score, 4.61, ahead of GPT-Image-1 (4.43), SymbOmni (4.35), and BAGEL (3.89), with favorable user assessments of instruction alignment, input-structure preservation, and satisfaction.It scores highest for Task Description Alignment (4.72), Input-Structure Fidelity (4.63), and Overall Satisfaction (4.62); GPT-Image-1 scores higher on Photorealism (4.68 vs. 4.50) and Visual Quality (4.64 vs. 4.56). Inquiry and downstream execution increase agent-side token consumption and reasoning time.Source: §A.8 · pp. 37–38, §A.9 · p. 38

Related papers

Loading 2609.16057v1…