EvoGen-Harness: Learning Where and How to Evolve Image-Generation Harnesses

1 Introduction

• Existing approaches often optimize a predefined adaptation dimension, limiting where failures can be corrected; visual feedback alone does not identify which persistent change is warranted.The same visible failure may reflect a harness weakness or an unlucky stochastic sample, making brute-force modification expensive and unreliable.Source: §1 · p. 2
• EvoGen-Harness is a generator-agnostic framework that evolves persistent external responsibilities while keeping the image generator frozen.Its practical responsibility set comprises Policy, Tools, Skills, Middleware, and Memory, and is not presented as an exhaustive taxonomy.Source: §1 · pp. 2–3
• Trace aggregates evidence across stochastic executions, uses failure attribution to prune candidate edits, and re-attributes residual failures to guide evolution across responsibilities.No-Patch avoids unnecessary updates, while held-out validation prevents harmful changes from persisting.Source: §1 · p. 3
• +0.2633, +0.0720, and +0.0752 over the strongest evaluated baselines on GenEval2, T2I-CompBench++, and WISE, respectively.The framework also achieves 87.9–91.4% attribution recall, 94.8% No-Patch accuracy, and 1.9% regression.Source: §1 · p. 3

2.1 Prompt Optimization and Visual Feedback

• Prompt-optimization methods use learned or searched prompts and visual feedback, but primarily optimize the prompt or current generation.Examples include RePrompt, VisualPrompter, and IterComp.Source: §2.1 · p. 3

2.2 Agentic and Self-Evolving Image Generation

• Agentic T2I systems add planning, tools, routing, and reusable experience, yet generally predefine what is improved.Examples span layout planning, regional decomposition, generator selection, and workflow or skill evolution.Source: §2.2 · p. 3

2.3 Agent Harness Evolution

• Prior harness-evolution work establishes a broader agentic paradigm; EvoGen-Harness targets stochastic image generation with trajectory-relative evidence and progressive re-attribution across responsibilities.The cited approaches include observability-driven harness evolution, weakness mining with regression validation, and on-demand task-adaptive harnesses.Source: §2.3 · pp. 3–4

3 Method

• EvoGen-Harness evolves a persistent external harness rather than modifying image-generator parameters.Trace identifies which harness component should evolve from visual execution evidence and iteratively refines it through feedback-driven updates.Source: §3 · p. 4

3.1 Multi-Responsibility Visual Harness

• The evolution space comprises Policy, Tools, Skills, Middleware, and Memory, encoding visual requirements, capabilities, reusable procedures, run-time orchestration, and cross-task experience.The harness is executed under multiple stochastic conditions, such as different random seeds.Source: §3.1 · p. 4

3.2 Trajectory-Relative Harness Attribution

• Trace aggregates constraint-level outcomes across stochastic executions because one failed image alone is insufficient evidence for a persistent update.A fixed LLM Harness Localizer scores plausible responsibilities and No-Patch; No-Patch is appropriate when stochasticity, verifier uncertainty, or insufficient evidence better explains the failure.Source: §3.2 · pp. 4–5

3.3 Trace: Attribution-Guided Progressive Evolution

• Trace explores localized candidate edits, with a responsibility-conditioned proposer restricted to the selected responsibility, and compares each candidate with its parent under matched stochastic conditions.Matched conditions make the measured score difference reflect the edit rather than random generation variation.Source: §3.3 · p. 5
• Candidate ranking penalizes preservation regression and execution cost; after an accepted edit, Trace re-localizes residual failures and can continue across responsibilities.The path persists only if it improves triggering and held-out cases while keeping preservation regression below tolerance; otherwise, the original harness is restored.Source: §3.3 · p. 5

4.1 Experimental Settings

• The evaluation uses official protocols on GenEval2, T2I-CompBench++, and WISE, with FLUX.1-dev frozen as the generator and separate models for localization, editing, and visual verification.Overall GenEval2 performance uses Soft-TIFA GM; the other reported measures include T2I-CompBench++ compositional metrics and WISE WiScore.Source: §4 · p. 6, §4.1 · pp. 6–7

4.2 Main Results

• 0.7089 Soft-TIFA GM on GenEval2, 0.6807 (+0.0720) on T2I-CompBench++, and 0.6780 (+0.0752) on WISE show gains across fine-grained constraints, compositions, and world-knowledge consistency.EvoGen-Harness outperforms the strongest standalone T2I model on GenEval2, leads in seven of eight T2I-CompBench++ dimensions, and is best across all six WISE domains.Source: Table 1 · p. 6, §4.2 · p. 7, Table 2 · p. 7

4.3 Generalization and Efficiency

• +0.4537, +0.3750, and +0.4674 GenEval2 gains over VisualPrompter on FLUX.1-dev, Qwen-Image, and Janus-Pro accompany positive gains for every reported visual skill.At similar model-call budgets, EvoGen-Harness scores 0.7089 versus OctoT2I’s 0.2895, though OctoT2I is faster; versus VisualPrompter, it is both faster (42.6 s vs. 61.8 s) and more accurate (0.7089 vs. 0.2552). Trace retains 0.7089 versus 0.7112 for full-space search while evaluating 8 rather than 20 candidate edits.Source: Table 4 · p. 8, §4.3 · p. 8, Table 5 · p. 8

4.4 Attribution and Evolution Analysis

• 87.9–91.4% attribution recall across five responsibilities and 94.8% No-Patch accuracy indicate localization of persistent causes while avoiding unnecessary evolution.Accepted updates span all five responsibilities, with update shares of 13.0–27.8% and positive held-out gains of +0.089 to +0.182; successive updates repair different constraints while the generator stays frozen.Source: §4.4 · p. 9, Table 6 · p. 9, Figure 4 · p. 10

4.5 Ablation Study

• Removing trajectory aggregation lowers GenEval2 from 0.7089 to 0.6580, removing attribution guidance lowers it to 0.6070, and disabling progressive re-localization lowers it to 0.6250.Removing No-Patch and safe validation raises regression from 1.9% to 10.7% and 13.9%, respectively; the full Trace reaches 0.7089 GenEval2 and +0.140 held-out gain with 1.9% regression.Source: Table 7 · p. 10, §4.5 · p. 10

• EvoGen-Harness provides a generator-agnostic framework for evolving multiple external harness responsibilities around frozen image generators.It moves beyond adaptation confined to a predefined dimension without modifying generator parameters.Source: §5 · p. 11
• TRACE guides search with trajectory-relative visual evidence and coordinates updates through re-attribution and regression-safe validation.Source: §5 · p. 11
• Across GenEval2, T2I-CompBench++, and WISE, the framework consistently improves generation quality while achieving reliable attribution, low regression, and gains across frozen T2I backbones.The paper presents selective evolution of the external harness as a practical path toward more capable, self-improving image-generation systems.Source: §5 · p. 11

A Evaluation Metrics

• Attribution is assessed with responsibility-level precision, recall, macro-averaged F1, and class-wise correct-attribution fractions.No-Patch is evaluated separately using its accuracy among ground-truth No-Patch examples.Source: A Evaluation Metrics · p. 14
• Held-out Gain compares an accepted update with its parent on a disjoint held-out set under matched stochastic conditions.A positive value means the update improves previously unseen cases, not only the triggering example.Source: A Evaluation Metrics · p. 14
• Regression is the percentage of previously successful preservation cases that become unsuccessful after a persistent update.The preservation set contains cases successfully handled before evolution.Source: A Evaluation Metrics · p. 14
• Responsibility statistics report each responsibility’s share of committed updates and the fraction of its proposed updates retained after validation.These statistics describe the distribution of evolution, not optimization objectives.Source: A Evaluation Metrics · p. 15
• Search efficiency is measured by average per-prompt inference time, generator and LLM/VLM calls, and candidate harness modifications evaluated per task.The benchmark evaluations use official metrics and protocols; the auxiliary metrics analyze Trace’s attribution and persistent evolution.Source: A Evaluation Metrics · pp. 14–15

B Implementation Details

• Evolution uses FLUX.1-dev with GPT-4.1 for localization and proposal, while visual verification and the deterministic execution controller remain fixed.Benchmark evaluation uses official releases and pipelines; internal stochastic executions are separate, and benchmark labels and scores are not exposed to the evolution components.Source: B Implementation Details · p. 15
• The evolution, held-out validation, and preservation splits contain 500, 100, and 100 prompts, respectively, and are disjoint from final benchmark evaluation sets.Exact and near-duplicate benchmark prompts are filtered out, and the evolved harness is frozen before benchmark testing.Source: B Implementation Details · pp. 15–16
• The five harness responsibilities are independently editable persistent records, and edits cannot directly change other responsibilities or the frozen generation system.The representation covers Policy, Tools, Skills, Middleware, and Memory; the examples describe editable state rather than generator parameters.Source: B Implementation Details · p. 16, Table 8 · p. 16
• Localized edits modify only the selected responsibility, with invalid changes to another responsibility, the generator, or the base controller rejected.Examples include hard cardinality constraints for Policy and a targeted retry after unresolved verification for Middleware; other responsibilities use the same localized interface.Source: B Implementation Details · pp. 16–17

B.1 Trace: Progressive Harness Evolution

• Trace retains at most r responsibilities and M candidate edits per responsibility at each step, then re-localizes residual failures after accepted edits.This bounds candidate evaluations over L steps while allowing the evolution path to traverse different responsibilities; held-out and preservation validation determine whether the update is retained.Source: §B.1 · pp. 17–18, Table 9 · p. 18

B.2 Search Efficiency Analysis

• Trace retains a Top-r localization frontier and progressively re-localizes residual failures, unlike full-space search or Hard Top-1’s single-responsibility search.Evolution-time cost is incurred when a persistent update is triggered and is amortized over subsequent requests, rather than included in per-prompt inference cost.Source: §B.2 · pp. 18–19
• Full-space search achieves only a marginally higher score than Trace (0.7112 vs. 0.7089), but requires substantially more candidate evaluations and generator calls.Source: §B.2 · p. 18
• Hard Top-1 is the most efficient strategy, but its score of 0.6815 is lower than Trace’s.The comparison holds the frozen generator, proposer, candidate and stochastic execution budgets, maximum depth, and matched-evaluation protocol constant.Source: §B.2 · p. 18

B.3 Robustness to Compound Failures

• The paired-failure test injects two persistent perturbations among all ten responsibility pairs while holding the request, generator, stochastic conditions, and other responsibilities fixed.This setup isolates interacting persistent causes with exact ground-truth attribution.Source: §B.3 · p. 19
• When two persistent failure causes are present simultaneously, Trace’s initial Top-3 frontier covers 94.3% of ground-truth causes on average, and progressive re-localization recovers both causes in 88.9% of compound cases.Source: §B.3 · pp. 19–20
• Sensitivity analysis finds diminishing performance gains as the attribution frontier grows, attribution gains saturating with more stochastic executions, and limited benefit from deeper evolution at higher cost.A larger frontier initially improves GenEval2 performance, while increasing K improves attribution reliability and increasing L improves held-out performance.Source: Figure 5 · p. 20
• Repair success reaches 83.5%, while Skills + Middleware is the most challenging pair at 79.6%.The gap between cause recovery and repair success shows that identifying both responsibilities does not always fully resolve the visual failures.Source: §B.3 · p. 20

B.4 Sensitivity Analysis

• The sensitivity analysis varies attribution frontier size r, stochastic executions K, and maximum evolution depth L.Source: §B.4 · p. 20
• Performance gains become marginal beyond moderate search, trajectory-evidence, and evolution-depth budgets.The study uses r=3, K=4, and L=3 as defaults for a favorable balance between evolution quality and computational cost.Source: §B.4 · p. 20
• The default configuration is r=3, K=4, and L=3, balancing evolution quality and computational cost.Source: §B.4 · p. 20

B.5 Robustness to Different Proposal LLMs

• The comparison replaces only the responsibility-conditioned proposal operator, keeping the Localizer, generator, verifier, candidate budget, seeds, and validation protocol unchanged.Source: §B.5 · p. 20
• Qwen3.5-27B retains performance close to GPT-4.1, while GPT-5.5 provides a modest further improvement.Source: §B.5 · p. 20
• The results indicate effectiveness primarily comes from Trace’s attribution-guided, regression-safe evolution procedure rather than a particular proposal LLM.Source: §B.5 · p. 21

B.6 Additional Responsibility-Specific Evolution Examples

• Policy, Skills, Tools, and Memory updates address distinct persistent failures while leaving the image generator frozen.Examples strengthen counting constraints, revise spatial-composition procedures, correct capability knowledge for attribute binding, and update action–object experience.Source: §B.6 · p. 21, Figure 6 · p. 21

B.7 Additional Cross-Generator Examples

• EvoGen-Harness consistently improves three frozen T2I backbones, with representative qualitative examples complementing the quantitative results.Source: §B.7 · p. 22
• Each backbone receives its own evolved harness; the paper does not assume zero-shot transfer of one evolved harness across generators.Examples span counting, spatial placement, and multi-object relations.Source: §B.7 · p. 22, Figure 7 · p. 22

C Harness Localizer Prompt

• The fixed Harness Localizer receives trajectory-relative evidence and sanitized harness state, then scores five editable responsibilities and a NO-PATCH option.Source: C Harness Localizer Prompt · p. 23

Harness Localizer Prompt

• The Localizer performs attribution only; visual verification identifies what failed, while the Localizer identifies potentially explanatory harness responsibilities.Source: Harness Localizer Prompt · p. 23
• NO-PATCH receives higher scores when stochastic generation, ambiguous intent, verifier uncertainty, or insufficient evidence better explains failures.Source: Harness Localizer Prompt · p. 23
• The prompt directs attribution to the earliest responsibility-specific decision that could plausibly explain a failure and to consider repeated stochastic evidence.Source: Harness Localizer Prompt · p. 23