Internalizing Agent Experience into Diffusion Model Weights via On-Policy Context Distillation

1 INTRODUCTION

• Harness gains remain external to the fixed generator, requiring guidance to be retrieved and reapplied for each request.As memories and skills accumulate, relevant guidance becomes harder to retrieve, and repeated reasoning, verification, and generation incur costs.Source: §1 · p. 1
• The transfer interface retains each completed task’s original query and harness-generated prompt, capturing prompt-mediated gains but not verification or stochastic sample selection.These query–prompt pairs train the generator so the gains are available from the original query alone.Source: §1 · pp. 1–2
• D-OPCD uses the harness-produced prompt as privileged context for a teacher that supervises a student conditioned only on the original query.The student is trained at states sampled from its own denoising trajectories, internalizing prompt-mediated gains into the generator weights.Source: §1 · p. 2
• ASE-driven D-OPCD improves direct-generation performance across all four benchmarks, and the updated generator with a skill-free harness slightly exceeds the original skill-equipped agent on average.Resetting the internalized skills lets ASE evolve again and yield further improvements.Source: §1 · p. 2
• The paper frames harness adaptation and generator learning as co-evolution: after internalization, the harness can shed saturated skills and resume evolving around the updated model.ASE turns task experience into reusable skills that provide the query–prompt pairs for D-OPCD.Source: §1 · p. 2

2.1 AGENTIC TEXT-TO-IMAGE GENERATION

• Agentic text-to-image systems combine planning, generation, visual inspection, and prompt revision, while memory and skills let them reuse experience across requests.Examples include GEMS, MemoGen, and GenEvolve, which use memory, reused experience, or generation trajectories to improve agent decisions.Source: §2.1 · p. 2

2.2 TRAINING DIFFUSION MODELS

• Diffusion generators can be updated through supervised fine-tuning or evaluative signals, and recent self-distillation methods use teacher predictions along a student’s sampling trajectory.D-OPCD instead uses harness-produced prompts as privileged context for the teacher while the student receives the original query alone.Source: §2.2 · pp. 2–3, Figure 1 · p. 3

2.3 HARNESS–MODEL CO-EVOLUTION

• Experience-driven agent improvement turns successful or failed trajectories into reusable guidance, with related work coupling evolving skills to policy or harness–model updates.Here, the evolving harness constructs prompts for a separate image generator.Source: §2.3 · p. 3

3.1 HARNESS GAINS AT THE GENERATOR INTERFACE

• Harness-selected prompts compile prompt-mediated experience, while their associated images indicate effectiveness; internalization aims to transfer that advantage to generation from the original query alone.The harness may also use external tools, but the method focuses on prompt-mediated gains exposed at the generator interface.Source: §3.1 · pp. 3–4, §3.2.1 · p. 4

3.2.1 HARNESS INTERNALIZATION AS CONTEXT TRANSFER

• D-OPCD gives a teacher the original query and harness prompt, while a student receives only the query, addressing direct SFT’s input-retention and off-policy limitations.The teacher and student are evaluated along the student’s own denoising trajectory.Source: §3.2.1 · p. 4

3.2.2 ON-POLICY DISTILLATION OBJECTIVE

• Harness-selected prompts from completed, verified tasks form the context dataset; the student and privileged-context teacher compare predictions on states visited by the student.The image is used to select records, while the prompt supplies the privileged context.Source: §3.2.2 · pp. 4–5

3.3 SELF-EVOLVING HARNESS VIA SKILL EXTRACTION

• ASE processes task episodes into insights and updates reusable skills, with mature insights informing proposals to create, revise, merge, or retire skills.Accepted updates mark absorbed insights as reviewed, so later updates draw on new evidence.Source: §3.3 · p. 5, Figure 2 · p. 5

3.4 HARNESS–GENERATOR CO-EVOLUTION

• After internalization, the generator adopts the updated weights, the learned skill library is cleared, and the harness evolves new versions against the updated generator.The signed internalization gap compares the updated generator and original skill-equipped agent on evaluation tasks.Source: §3.4 · p. 6

4.1 EXPERIMENTAL SETUP

• The evaluation uses four benchmarks spanning compositional, knowledge-intensive, and reasoning-intensive generation, with Qeval results and a separate OOD test.The initial generator is Z-Image-Turbo; the harness starts with an empty skill library and evolves through ASE on Qtrain.Source: §4.1 · p. 6

4.2 MAIN RESULTS

• D-OPCD raises direct-generation average from 60.52 to 65.09; the skill-free and ASE-evolved harnesses raise the base-generator average to 79.28 and 81.83.Both harness configurations improve over direct generation on all four benchmarks.Source: §4.2 · p. 7, Table 1 · p. 7
• With the skill-free harness, the updated generator averages 82.33 and slightly exceeds the original skill-equipped agent, retaining much of the earlier skill benefit.The signed internalization gap is +0.50, though it varies across benchmarks.Source: §4.2 · p. 7, Table 1 · p. 7
• After ASE runs again, the updated agent averages 84.16 (+2.33); R2I-Bench instead declines after skill evolution.The renewed skills improve the updated agent on three benchmarks and raise its average above both the updated skill-free agent and the original skill-equipped agent.Source: §4.2 · p. 7, Table 1 · p. 7

4.3.1 COMPARISON WITH ALTERNATIVE TRAINING METHODS

• D-OPCD reaches a 65.09 four-benchmark average, 1.71 points above Vanilla SFT, and leads on all four benchmarks.Methods train on matched records collected by the evolved harness and are evaluated by direct generation from original queries.Source: §4.3.1 · p. 8, Table 2 · p. 8

4.3.2 ABLATION OF TEACHER CONDITIONING

• Matched query-plus-prompt teacher input averages 65.09, ahead of prompt-only (64.63) and shuffled prompts (61.05); shuffled prompts reduce scores on all four benchmarks.The student always receives the original query.Source: §4.3.2 · p. 8, Table 3 · p. 8

4.3.3 CONTEXT PROPERTIES AND MODEL INTERNALIZATION

• Context variants show mixed downstream effects: filtering leaves the mean nearly unchanged, while the diversity adapter lowers it.Filtering also changes training-set size and composition, so its effect cannot be attributed solely to task improvement; which contexts best support internalization remains open.Source: §4.3.3 · p. 9, Table 4 · p. 9

4.3.4 OUT-OF-DISTRIBUTION GENERALIZATION

• On T2I-CompBench++ OOD tests, GenEval- and GenEval2-trained checkpoints improve Color and 2D-Spatial scores, while WISE- and R2I-Bench-trained checkpoints remain close to base overall.The reported Color scores are 81.61 and 81.44 versus 77.45 for the base; 2D-Spatial scores are 36.19 and 37.33 versus 32.32.Source: §4.3.4 · p. 9, Table 5 · p. 9

• D-OPCD transfers prompt-mediated gains from an evolving text-to-image harness into the generator’s weights, retaining part of the harness’s benefit from the original query alone.Source: §5 · p. 9
• With ASE-generated skills, D-OPCD improves direct generation across all four benchmarks, and the updated generator with a skill-free harness slightly exceeds the original skill-equipped agent on average.Source: §5 · p. 9
• Resetting the old skills lets ASE evolve again and further improve performance, supporting the paper’s aim of continued harness–model co-evolution.Source: §5 · p. 9

• Figure 3 shows selected cases; Table 1 reports aggregate results over the full evaluation sets.Source: A QUALITATIVE EXAMPLES FOR THE MAIN… · p. 13, Figure 3 · p. 13
• Figure 3 contrasts outputs from the base generator and the D-OPCD-updated generator for four selected test queries.Source: Figure 3 · p. 13
• Within each generator group, the columns show direct generation, the skill-free harness, and evolved skills.Source: Figure 3 · p. 13
• The text below each row gives the original query, while harness runs may use selected generation prompts.Source: Figure 3 · p. 13

B DISCUSSION ON LIMITATIONS AND FUTURE WORKS

• D-OPCD raises four-benchmark direct-generation average from 60.52 to 65.09, still below the original skill-equipped harness’s 81.83; evolved prompts may not align well with the diffusion model’s latent space.The authors suggest that stronger joint adaptation may be needed for sustained co-evolution.Source: B DISCUSSION ON LIMITATIONS AND FUTURE… · p. 13
• Evidence for repeated co-evolution is limited to one base generator, one D-OPCD update, four benchmarks, and one subsequent skill-evolution round.The experiments do not establish outcomes across many update-and-reset cycles, other generators, or task distributions.Source: B DISCUSSION ON LIMITATIONS AND FUTURE… · p. 13
• The studied agent uses prompt construction, verification, and refinement without external knowledge search or reference-image tools, limiting the demonstrated scope.Tool-augmented experience may include information and visual evidence that a final text prompt cannot fully represent.Source: B DISCUSSION ON LIMITATIONS AND FUTURE… · p. 14

C.1 GEMS-BASED AGENT HARNESS DESIGN

• The harness selects Skills to form an initial prompt, decomposes the original query into fixed visual checks, and revises prompts using verification feedback and task-local history.It returns the prompt and image from the earliest attempt with the highest number of requirements satisfied.Source: §C.1 · p. 14

C.2 EPISODE CONSTRUCTION AND INSIGHT EXTRACTION

• Each completed task becomes one Episode recording task metadata, attempts, verification, selected Skills, retrieved Insights, and the returned outcome.Source: §C.2 · p. 14
• The learner retrieves a frozen working set of relevant Insights, and the manager proposes evidence-backed operations that are validated before updating the pool.Support and contradiction counts use distinct Episodes; maturity thresholds govern when Insights reach the Skill manager.Source: §C.2 · p. 15

C.3 SKILL UPDATES AND LIBRARY VERSIONING

• Skill evolution is triggered by accumulated mature, unreviewed Insights and review intervals; edits are constrained by a maximum of five active Skills and 4,500 characters per Skill.Committed edits record absorbed Insights, while a no-op completes review without changing the library.Source: §C.3 · pp. 15–16
• At inference, the agent routes each request to relevant Skills before generation, and records prompt–image pairs with harness and generator versions for D-OPCD.Source: §C.3 · p. 16

C.4 EXPERIMENTAL CONFIGURATION AND FROZEN REPLAY

• The agent uses GPT-5.6 Terra with medium reasoning effort for reasoning, Insight and Skill management, and in-task verification; evaluation leaves agent state unchanged.Tables 8 and 9 give shared and benchmark-specific ASE settings.Source: §C.4 · p. 16, Table 8 · p. 17, Table 9 · p. 17
• Task i’s generation attempt j uses seed si + j − 1, with si as the task’s base seed.Source: §C.4 · p. 17
• Skill-library selection uses Qhold, then the selected library is frozen for replay on Qtrain and evaluation on Qeval; replay returns query, prompt, and image records for eligible prompt pairs.Tasks replayed or evaluated do not update the persistent Skill library or Insight store.Source: §C.4 · p. 17

C.5.1 AGENT EXECUTION PROMPTS

• Figures 4–9 present the agent’s execution prompts in task order, with braced capitalized fields marking runtime substitutions.The optional cross-task Insight field is used during evolution but left empty in frozen skill-only replay and evaluation.Source: §C.5.1 · p. 17

C.5.2 HARNESS EVOLUTION PROMPTS

• Separate Insight and Skill manager prompts govern cross-task knowledge updates; Figures 10 and 11 show their decision instructions and runtime inputs.The operation schemas appear in Tables 6 and 7 rather than in the prompt figures.Source: §C.5.2 · p. 17, Figure 10 · p. 20, Figure 11 · p. 21
• The execution prompts route Skills, rewrite queries, decompose requirements, verify images, summarize task experience, and refine prompts after failed checks.Source: Figure 4 · p. 18, Figure 5 · p. 18, Figure 6 · p. 18

D.1 TRAINING DATA PREPARATION

• D-OPCD uses frozen replay records from the selected skill-equipped harness and trains on query–prompt pairs; the associated image determines eligibility but is not a loss target.Compared methods use the same subset after filtering normalized cases where prompt equals query.Source: §D.1 · p. 22

D.2 D-OPCD IMPLEMENTATION

• The student follows a four-state Euler rollout, while an EMA copy provides stop-gradient teacher predictions conditioned on the original query and privileged prompt.The mean squared velocity difference is minimized at the student’s visited states; the student and teacher share a frozen text encoder.Source: §D.2 · p. 22

D.3 BASELINE OBJECTIVES

• Vanilla SFT uses the agent-submitted image latent for flow-matching supervision, while Diffusion-DPO compares chosen and same-seed rejected images against a frozen reference.D-OPSD instead uses on-policy student states and an EMA teacher’s detached clean-output target, with image embeddings from Qwen3-VL-4B-Instruct.Source: §D.3 · p. 22

D.4 OPTIMIZATION AND CHECKPOINT SELECTION

• All methods train LoRA adapters on Z-Image-Turbo for 2,000 optimizer steps, with direct evaluation using eight sampling steps and guidance scale zero.Shared settings include rank 64, scale α = 128, 1024-pixel square images, and a constant learning rate of 10^-4.Source: §D.4 · p. 23
• Each checkpoint is evaluated on Qhold, and the benchmark-native held-out metric is used for checkpoint selection.Table 11 lists the reported optimizer step and held-out score by benchmark and method; its caption identifies the metrics but supplies no cell values.Source: §D.4 · p. 23, Table 11 · p. 23

E.1 TASK SET ALLOCATIONS

• GenEval, GenEval2, and WISE use approximately 50/15/35 train, hold, and evaluation allocations; R2I-Bench uses a 780-task experimental subset with its supplied split.Table 12 identifies these as allocations over the task pools actually used.Source: §E.1 · p. 23, Table 12 · p. 24
• The first three benchmark splits use seed 42, while GenEval2 and WISE preserve earlier task assignments and R2I-Bench uses its supplied split.Source: §E.1 · p. 24

E.2 REPLAY DATA CONSTRUCTION AND COUNTS

• Frozen replay on Qtrain yields one submitted query, prompt, and image record per task, with prompt-changed tasks used by all four training methods.Table 13 reports record counts before and after excluding normalized unchanged prompts; replay uses the selected skill-equipped harness version.Source: §E.2 · p. 24, Table 13 · p. 24

Improvement-filtered pairs

• The improvement filter retains task pairs when the selected harness image passes more checklist checks than the direct-query image, excluding missing or invalid verifier responses.Both images are assessed against the selected attempt’s checklist; the filter uses no new image generation or external benchmark scores, and differing seeds mean it selects observed improvements rather than estimating a seed-controlled prompt effect.Source: §E.3 · p. 24
• The improvement filter retains 43, 264, 319, and 314 pairs on GenEval, GenEval2, WISE, and R2I-Bench, respectively.Three retained R2I-Bench pairs have unchanged prompts but meet the image-level criterion.Source: §E.3 · p. 24

Diversity-adapted sampling

• The diversity adapter randomly selects one of four rewrite strategies when Skills apply, while preserving task requirements and varying how guidance is expressed.The strategies are Local edit, Restate, Integrate, and Separate; Figure 12 shows the shared rewrite template and Table 14 gives the instructions.Source: §E.3 · p. 24, Figure 12 · p. 25, Table 14 · p. 25

Paired task effectiveness

• Paired effectiveness compares same-seed images generated from the original query and harness prompt, scoring both against the original query and benchmark annotations.Table 15 defines the reported measures as score difference, relative percentage, and the share of tasks where the prompt image scores strictly higher; no outcome values are supplied here.Source: §E.3 · p. 25, Table 15 · p. 26

Context-text diversity

• Text diversity is measured with normalized character Vendi, within-set pairwise character TF–IDF cosine, and corpus Distinct-1, alongside mean prompt length.The measures are text-diversity proxies, not image-diversity scores; Table 16 states that higher Vendi and Distinct-1 and lower cosine indicate more varied contexts.Source: §E.3 · p. 26, Table 16 · p. 26
• In a separate matched-400 GenEval2 audit, the diversity adapter scored 0.7921 versus 0.7402 on normalized lexical Vendi and 0.3862 versus 0.3533 on Distinct-2.The audit used a different word-level feature space and sample protocol from Table 16; on GenEval, word-level differences were small and mixed.Source: §E.3 · p. 26

E.4 EVALUATION METRICS AND PROTOCOLS

• In-distribution evaluation generates one image per Qeval task and reports each benchmark’s native metric under its official protocol.GenEval uses Overall, GenEval2 Soft-TIFA AM, WISE WiScore, and R2I-Bench R2I-Score; Qhold checkpoint evaluation uses the same metrics.Source: §E.4 · p. 27
• T2I-CompBench++ is evaluated separately on 2,100 validation prompts, with three images per prompt and category-level scores.This OOD set is not used for skill evolution or model or harness selection; each non-base row reports one source-specific D-OPCD checkpoint.Source: §E.4 · p. 27

F USE OF AI ASSISTANTS

• AI assistants supported prompt planning and refinement, benchmark-native scoring, and manuscript wording, but did not originate research questions or produce reported results.The authors reviewed and approved the experimental design, analyses, results, and manuscript text.Source: F USE OF AI ASSISTANTS · p. 27