PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoRef: Harness Optimization for Agentic Multi-Reference Image Generation

Yuta Oshima, Ku Onoda, Yusuke Iwasawa, Masahiro Suzuki, Yutaka Matsuo, Hiroki Furuta

arXiv:2609.35530v1cs.CVcs.AI

TL;DR

Multi-reference image generation can omit or duplicate referenced subjects or make them appear pasted into a scene. AutoRef optimizes the harness around frozen models, and AutoRef-Harness improves FLUX.2 [klein] 4B while transferring across settings.

  • Problem

    Multi-reference generation remains challenging because models may omit or duplicate referenced subjects or make them appear pasted into the scene.

    Source: §1 · p. 2

  • Method

    AutoRef iteratively rewrites the harness around frozen models, separating proposal feedback from candidate selection and continuing from a beam of top-ranked harnesses.

    Source: §1 · p. 2, §8 · pp. 9–10

  • Results

    AutoRef-Harness transfers across reference counts, benchmarks, evaluators, generators, and reasoning models; on held-out four-reference MultiBanana tasks, FLUX.2 [klein] 4B improves from 5.72 to 7.37 and matches or exceeds Nano Banana Pro and GPT-Image-1.5.

    Source: §1 · p. 2, §8 · p. 10

  • Takeaways & Limitations

    The results suggest that how frozen models are used is itself worth optimizing.

    Source: §8 · p. 10

    Harness optimization cannot exceed what the generator can produce, so better selection does not help when no draft is acceptable, as with Qwen-Image-Edit-2511 at five references.

    Source: §7 · p. 9

Abstract

from arXiv · show

Recent image generation models can take multiple reference images as input and combine them into a new image. However, multi-reference image generation remains challenging: models may omit or duplicate subjects from the references, or produce images in which multiple subjects appear unnaturally pasted. Recent work has proposed image generation agents that combine image generation models, reasoning models, and a harness, which is an executable program that specifies how reference images are interpreted, how generation is performed, how outputs are diagnosed, and how the final image is selected. In multi-reference generation, however, references play different roles and outputs must satisfy many criteria at once, such as fidelity to each reference and the naturalness of the whole image, so many parts of the harness could be improved, from how references are processed to how outputs are diagnosed. This makes it hard to predict which changes will improve performance and by how much, and good harnesses difficult to design by hand; indeed, human-written harnesses vary widely in performance. We therefore propose AutoRef, which optimizes the harness automatically while keeping both models frozen: a coding agent iteratively rewrites the harness code. AutoRef separates the tasks whose feedback informs proposals from the tasks used to select candidates, and continues the search from a beam of the top-ranked harnesses on the selection tasks. Using this procedure, we discover AutoRef-Harness, which improves the open-weight FLUX.2 [klein] 4B from 5.72 to 7.37 on held-out four-reference tasks of the MultiBanana benchmark, matching or exceeding proprietary models including Nano Banana Pro and GPT-Image-1.5. Without re-optimization, the same harness also improves results when the generator, number of references, benchmark, evaluator, or reasoning model differs from those used in the search.

1 INTRODUCTION

Multi-reference image generation is difficult because references can play different roles and generated subjects may be omitted, duplicated, or pasted unnaturally. AutoRef automatically optimizes the harness while keeping the reasoning and image generation models frozen.

1 INTRODUCTION

  • Multi-reference generation lets users specify people, objects, clothing, backgrounds, and styles separately, but models may omit or duplicate subjects or make them look pasted into the scene.The passage notes applications in advertising, virtual try-on, and content creation.Source: §1 · p. 2
  • Harness design is difficult because references can specify different roles and outputs must balance fidelity to each reference with the naturalness of the whole image.The effects of potential harness changes are hard to predict, and human-written harnesses vary widely in performance.Source: §1 · p. 2
  • AutoRef separates tasks that inform harness proposals from tasks that select candidates, then continues from the top-ranked harnesses in an iterative beam search.The generator and reasoning model remain frozen while a coding agent rewrites the harness.Source: §1 · p. 2
  • 5.72 to 7.37: AutoRef-Harness improves FLUX.2 [klein] 4B on held-out four-reference MultiBanana tasks, matching or exceeding Nano Banana Pro and GPT-Image-1.5.The harness also improves results without re-optimization when the generator, number of references, benchmark, evaluator, or reasoning model changes.Source: §1 · p. 2

2 RELATED WORK

Related work spans multi-reference generation and automatic agent optimization. AutoRef addresses the limited gains of fixed agent workflows by automatically optimizing harnesses from task feedback while keeping the generator frozen.

Multi-Reference Image Generation

  • Multi-reference image generation has advanced through flexible reference-conditioned models and scaling training data or fine-tuning, while simple agentic refinement has yielded limited gains on MultiBanana.The authors argue that simply wrapping a strong generator in a fixed workflow is insufficient, motivating automatic harness optimization with the generator frozen.Source: §2 · p. 2

Automatic Optimization of Agentic Systems

  • Automatic agent optimization covers prompts, pipelines, workflows, and executable code, but cited held-out selection methods tune prompts within fixed programs.Meta-Harness lets a coding agent revise a harness around a frozen model but evaluates candidates on the same tasks that provide feedback; AutoDesign applies harness optimization to paper-to-poster generation.Source: §2 · p. 3

3 PRELIMINARIES

The paper formalizes a multi-reference task, a harness that runs frozen reasoning and image models, and an evaluator-based performance objective. Optimization changes the executable code rather than model parameters.

Multi-Reference Image Generation

  • A task pairs a user instruction with a set of reference images, and running a harness yields an image and an execution trajectory.The harness specifies reference interpretation, prompt construction, candidate generation and evaluation, and output selection.Source: §3 · p. 3

Harness Optimization

  • Harness optimization targets expected evaluator-scored image quality over multi-reference tasks while keeping the reasoning and image-generation model parameters frozen.Changes can therefore affect prompting, generation, evaluation, selection, and control flow through the executable code.Source: §3 · p. 3

Harness Search Loop

  • A coding-agent proposer uses the current harness and accumulated artifacts, including scores and execution trajectories, to diagnose failures and propose an updated harness.Each proposed harness is evaluated on search tasks, and its implementation, scores, and trajectories are added to the history.Source: §3 · p. 3

4 AUTOREF

AutoRef adapts harness optimization to image generation, where noisy visual evaluation makes failures hard to diagnose and repeated search can overfit. It separates proposal feedback from candidate selection and continues through beam search.

4 AUTOREF

  • Visual evaluation can be noisy, failures hard to diagnose from scalar rewards, and repeated optimization on limited examples can overfit both tasks and evaluator.These properties motivate task separation and beam search for perceptual, non-verifiable image generation.Source: §4 · pp. 3–4
  • Figure 2 illustrates one iteration: the proposer writes four harnesses, training feedback is available for all candidates, and validation keeps two without revealing their scores.The survivors, not their parents, form the next beam.Source: Figure 2 · p. 4
  • AutoRef separates proposal and selection tasks and keeps the top-ranked validation candidates as parents, addressing overfitting risks from noisy visual evaluation.The validation scores and artifacts are hidden from the proposer; training-side scores, rationales, trajectories, and visual artifacts remain available as feedback.Source: §4 · p. 4
  • At each iteration, a coding-agent proposer uses the current beam and search history to write candidates; validation ranking selects the next beam.The proposer is told which candidates survived, but not their validation scores, and the history retains training-side evidence from selected and unselected candidates.Source: §4 · p. 4

5 THE OPTIMIZED AUTOREF-HARNESS

AutoRef-Harness chains reference-specific prompting, structurally distinct drafts, complaint-directed revision, and failure-aware selection. The stages specialize the harness for the different roles references play and for the need to produce a coherent scene.

5 THE OPTIMIZED AUTOREF-HARNESS

  • AutoRef-Harness draws three images: it compares drafts A and B, revises the better draft using complaints, then selects between that draft and C.Its steps are specialized for multiple references and chained; the components were added in iterations that raised validation score.Source: §5 · pp. 4–5

5.1 REFERENCE-GROUNDED PROMPTING

  • Each prompt assigns requested elements to their respective references and excludes content the user did not request.The reasoning model reads the instruction and all references, then writes the prompt rather than passing the raw instruction directly to the generator.Source: §5.1 · p. 5

5.2 STRUCTURALLY DIVERSE DRAFTS

  • Draft A describes the scene subject by subject, while draft B uses a designated background or style reference as a canvas for jointly rendering the other subjects.If no background or style reference is designated, draft B resamples draft A’s prompt.Source: §5.2 · p. 5

5.3 COMPLAINT-DIRECTED REVISION

  • The reasoning model lists up to five concrete complaints about the better of drafts A and B, names the reference each concerns, and revises the prompt to address them.Draft C uses the revised prompt, or resamples the winner’s prompt if there are no complaints.Source: §5.3 · p. 5

5.4 FAILURE-AWARE SELECTION

  • Selection first favors the draft with fewer hard failures; on a tie, the challenger must win the reasoning model’s pairwise judgment in both presentation orders.Hard failures include missing references, extra or duplicated subjects, and a wrong background.Source: §5.4 · p. 5

6 EXPERIMENTS

On held-out four-reference MultiBanana tasks, AutoRef-Harness improves FLUX.2 [klein] 4B and produces more coherent reference integrations. Its gains also transfer across reference counts, benchmarks, evaluators, reasoning models, and generators, while comparisons and ablations examine the harness and its components.

6.2 MAIN RESULTS

  • AutoRef-Harness improves FLUX.2 [klein] 4B on held-out four-reference MultiBanana tasks, outperforming evaluated open models and remaining competitive with proprietary models.The same harness also improves FLUX.2 [klein] 9B and Qwen-Image-Edit-2511 without updating the image generator or reasoning model.Source: Table 1 · p. 6, §6.2 · pp. 6–7
  • In qualitative examples, AutoRef-Harness preserves each reference and integrates it naturally, while baselines often omit, duplicate, misplace, or paste references unnaturally.Examples include base FLUX.2 [klein] 4B duplicating a hawk and misplacing a woman, and copy-and-paste-like results from Nano Banana Pro and Qwen-Image-Edit-2511.Source: Figure 3 · p. 7, §6.2 · p. 7

6.3 TRANSFERABILITY OF AUTOREF-HARNESS

  • The harness improves FLUX.2 [klein] 4B on unseen three- and five-reference settings, though Qwen-Image-Edit-2511 is an exception at five references.The reported exception occurs when the generator itself fails and the harness cannot compensate.Source: Table 2 · p. 6, §6.3 · p. 7
  • On unseen OmniContext, scored by GPT-4.1 rather than the search evaluator, AutoRef-Harness again improves FLUX.2 [klein] 4B and remains competitive with proprietary image models.The benchmark and evaluator both differ from those used during search, with no benchmark-specific harness modifications.Source: §6.3 · p. 7, Table 3 · p. 8
  • With Qwen3-VL-32B replacing GPT-5.5, AutoRef-Harness improves FLUX.2 [klein] 4B from 5.72 to 6.90 on the held-out test split; with GPT-5.5, it scores 7.37.The harness structure is unchanged across reasoning models.Source: §6.3 · p. 8, Figure 4 · p. 8

6.4 COMPARISON WITH HARNESSES AND HARNESS SEARCH

  • AutoRef-Harness achieves the highest held-out performance among the compared human-written harnesses, while AutoRef’s second-best harness beats the final harnesses from Meta-Harness and Greedy Search.Idea2Img underperforms despite using three times AutoRef-Harness’s generation budget; AutoRef’s final harness also draws fewer images per task than the alternative search methods’ final harnesses.Source: §6.4 · pp. 8–9, Figure 4 · p. 8

6.5 ABLATION STUDY

  • Removing structurally diverse drafts, complaint-directed revision, or failure-aware selection lowers the average score, and no partial combination matches the full harness.Grounding only scores 6.91 with one image; selection only scores 6.15, while the selector adds 0.44 within the full harness (7.37 vs. 6.93).Source: §6.5 · p. 9, Table 4 · p. 9

6.6 HUMAN EVALUATION

  • Human raters prefer AutoRef-Harness to base FLUX.2 [klein] 4B 70% of the time and find it competitive with Nano Banana Pro (46% vs. 40%).It also wins more often than it loses against FLUX.2 [klein] 9B (64% vs. 24%) and Seedream 4.5 (63% vs. 27%).Source: §6.6 · p. 9

7 DISCUSSION AND LIMITATIONS

AutoRef-Harness is limited by the capabilities of its frozen generator, and the authors identify a setting where better selection does not help. They also report transfer beyond the evaluator used in search.

7 DISCUSSION AND LIMITATIONS

  • Figure 5 reports win, tie, and loss rates for AutoRef-Harness against each baseline in pairwise human evaluation.Source: Figure 5 · p. 9
  • When no draft is acceptable, as with Qwen-Image-Edit-2511 at five references, better selection does not help because the harness cannot exceed its frozen generator’s capabilities.The authors also note that the harness transfers to OmniContext with its GPT-4.1 evaluator, despite search maximizing one VLM evaluator’s score.Source: §7 · p. 9
  • The authors identify stronger VLMs, coding agents, and richer evaluators—including VLMs combined with segmentation models—as directions for exploration.Source: §7 · p. 9

8 CONCLUSION

AutoRef optimizes the harness program while keeping the image generator and reasoning model frozen, using separated proposal and selection tasks with iterative beam search. The resulting harness transfers across several settings, while remaining constrained by the underlying models and evaluators.

8 CONCLUSION

  • AutoRef keeps the image generator and reasoning model frozen and changes only the program that combines them.It separates tasks that inform proposals from tasks used to select candidates, continuing from top-ranked harnesses through iterative beam search.Source: §8 · pp. 9–10
  • AutoRef-Harness makes FLUX.2 [klein] 4B competitive with proprietary models on MultiBanana and transfers across reference counts, benchmarks, evaluators, generators, and a reasoning model.The authors also report gains alongside fine-tuning and conclude that how frozen models are used is worth optimizing.Source: §8 · p. 10
  • The authors report using generative AI tools for methodology feedback, implementation, analysis support, figure and image creation, code, and paper writing, and state that two authors verified and tested generated code and writing.Source: AI USE STATEMENT · p. 10
  • The approach optimizes only the harness, so it does not update the underlying models and inherits limitations and biases from the models and evaluators.The authors also note possible misuse of improved multi-reference generation for misleading or unauthorized synthetic content.Source: ETHICS STATEMENT · p. 10
  • The authors release the implementation, AutoRef-Harness, data splits, and pinned model revisions, with evaluations on publicly available benchmarks.Source: REPRODUCIBILITY STATEMENT · p. 10

A IMPLEMENTATION DETAILS

The paper situates harness optimization among test-time scaling and agent optimization, then describes AutoRef’s iterative code-search procedure. Its implementation uses pinned model configurations and validation-ranked candidates while keeping validation artifacts from the proposer.

A IMPLEMENTATION DETAILS

  • The implementation pins the reasoning-model snapshot and open-weight model revisions, uses greedy evaluator decoding, and specifies API settings for proprietary models.The proposer is Claude Fable 5.1, run through the Claude Code CLI; Table 5 lists model identifiers.Source: A IMPLEMENTATION DETAILS · p. 17, Table 5 · p. 17

B EXTENDED RELATED WORK

  • The paper frames iterative generation, evaluation, and revision as test-time scaling, but optimizes the harness to find a more effective and computationally efficient inference procedure.Source: B EXTENDED RELATED WORK · pp. 17–18
  • Unlike human-written or trained image-generation agents, AutoRef searches over harness code while keeping all models frozen.Source: B EXTENDED RELATED WORK · p. 18

C THE AUTOREF ALGORITHM

  • At each iteration, a coding-agent proposer uses the current beam and accumulated training artifacts to write candidate harnesses, which are run on training tasks and added to the history.The next beam consists of the B candidates with the highest validation score; validation scores and outputs are not shown to the proposer.Source: C THE AUTOREF ALGORITHM · p. 18
  • With T = 5, K = 4, and B = 2, AutoRef evaluates 20 candidates, and AutoRef-Harness is the final-beam member with the highest validation score.Source: C THE AUTOREF ALGORITHM · p. 18

D HOW THE HARNESS CHANGED ACROSS ITERATIONS

Across five iterations, AutoRef reaches a higher validation score than the two search baselines shown. The section traces the harness changes and describes how the proposer’s prompts, information access, and candidate rules shape the search.

D HOW THE HARNESS CHANGED ACROSS ITERATIONS

  • The retained lineage progresses from reference-grounded prompting to diverse drafts, complaint-directed revision with hard-failure checks, and canvas-anchored structural diversity.The iteration scores reported along this lineage include 6.65, 6.81, 7.30, and 7.32.Source: D HOW THE HARNESS CHANGED ACROSS… · p. 19
  • Edits left flagged failures on 8 of 12 tasks, while single-light re-rendering lowered a draft’s score from 6.16 to 5.91; these directions were dropped.Other unretained variants included subject crops, grounding only after failure, and structured prompt-framing ladders.Source: D HOW THE HARNESS CHANGED ACROSS… · p. 19
  • After five iterations, AutoRef reaches JDval 7.32, versus 7.15 for Greedy Search and 6.74 for Meta-Harness.Source: Figure 7 · p. 20, D HOW THE HARNESS CHANGED ACROSS… · p. 20

E PROMPTS FOR PROPOSER CODING AGENT

  • The proposer starts a fresh coding-agent session at every iteration and receives a fixed system prompt plus an iteration-specific prompt describing the current beam and permitted logs.Source: E PROMPTS FOR PROPOSER CODING AGENT · p. 20
  • The proposer can inspect earlier candidates’ training-task code, scores, rationales, trajectories, and images, but cannot access validation or held-out test results.The next bases are selected on tasks outside the training set that the session cannot see.Source: E PROMPTS FOR PROPOSER CODING AGENT · pp. 20–21
  • The harness interface exposes calls to the frozen reasoning and image models, while tracking every call made during a run.Source: E PROMPTS FOR PROPOSER CODING AGENT · p. 20
  • Candidate rules discourage parameter-only variants and task-specific hints, instead asking for broadly applicable changes to harness mechanisms.Source: E PROMPTS FOR PROPOSER CODING AGENT · p. 21
  • The iteration prompt specifies the task, iteration, candidate evaluation workload, and available logs; its example requests evaluation on 48 training tasks.Logs include prior results, the frontier, per-harness scores and traces, reports, and beam/adoption records.Source: E PROMPTS FOR PROPOSER CODING AGENT · p. 21

F THE FLOW OF AUTOREF-HARNESS

AutoRef-Harness interprets references, creates and compares differently structured drafts, then revises the preferred draft from explicit complaints. The section also contrasts human-written harnesses and search strategies.

F THE FLOW OF AUTOREF-HARNESS

  • Both drafts receive a hard-failure check before pairwise selection; complaints about the winner guide a third draft, which is returned only if it wins.The checks cover missing or extra subjects and a wrong background.Source: F THE FLOW OF AUTOREF-HARNESS · pp. 22–23
  • The harness identifies a background or setting reference, then builds scene-first and canvas-anchored prompts that structure the scene differently.When a scene reference is identified, it is placed first as the canvas; otherwise, the second draft resamples the first prompt.Source: F THE FLOW OF AUTOREF-HARNESS · p. 22

G.1 HUMAN-WRITTEN HARNESS BASELINES

  • Human-written baselines span one-shot reranking, verifier-driven loops, iterative prompt refinement, and memory-based self-refinement; Idea2Img’s official setup uses nine images versus AutoRef-Harness’s three.The described IPR reimplementation returns its last image, while GEMS returns the image satisfying the most questions.Source: §G.1 · p. 23

G.2 HARNESS SEARCH BASELINES

  • Meta-Harness ranks proposals on the same tasks whose feedback informs them, whereas AutoRef separates proposal and selection tasks; Greedy Search keeps only one selection-ranked parent.Meta-Harness runs five iterations here, compared with 20–40 in the original protocol.Source: §G.2 · p. 23

H.1 DETAILED MULTIBANANA RESULTS

The detailed results cover evaluator criteria, component ablations, compatibility with fine-tuning, OmniContext breakdowns, a generator limitation, and selected qualitative comparisons.

H.1 DETAILED MULTIBANANA RESULTS

  • The MultiBanana evaluator scores instruction alignment, reference consistency, background–subject match, physical realism, and visual quality; Table 6 reports their averages for four-reference held-out tasks.The reported Avg. is the mean of those five metrics.Source: §H.1 · p. 24, Table 6 · p. 24

H.2 COMPONENT ABLATION BY TASK TYPE

  • Removing any one of the three components lowers scores on every task type, while selection without reference-grounded prompting scores below all three leave-one-out rows on every type.Table 9 reports component ablations on the held-out test split by task type.Source: §H.2 · p. 24, Table 9 · p. 25

H.4 DETAILED OMNICONTEXT RESULTS

  • The OmniContext breakdown reports prompt following, subject consistency, and their geometric mean across single, multiple, and scene task types.Tables 10 and 11 provide the detailed scores behind Table 3.Source: §H.4 · p. 25, Table 10 · p. 26, Table 11 · p. 26

H.3 COMPATIBILITY WITH FINE-TUNING

  • 5.90 with AutoRef-Harness on the DyRef-trained Qwen-Image-Edit-2511 model exceeds 4.91 with DyRef alone; the harness alone scores 5.69.The authors describe harness and model-weight optimization as complementary and combinable.Source: §H.3 · p. 24, Figure 9 · p. 26

I WHERE THE HARNESS CANNOT HELP

  • At five references, Qwen-Image-Edit-2511 changes from 1.89 to 1.87 with AutoRef-Harness, while all three drafts fail hard checks on 87 of 96 tasks (91%).The harness can select only among images produced by the frozen generator; the passage suggests reducing the number of references composed at once as a likely way to overcome this limit.Source: I WHERE THE HARNESS CANNOT HELP · p. 27, Figure 10 · p. 27

J FURTHER QUALITATIVE COMPARISONS

  • Selected held-out examples span composition, attribute transfer, background and style, unseen reference counts, and OmniContext; they emphasize large score gaps rather than a random sample.Across settings, the harness removes failures where a requested reference is dropped or used in the wrong role.Source: J FURTHER QUALITATIVE COMPARISONS · p. 28

Related papers

Loading 2609.35530v1…