PaperlayerView exact revision on arXiv

Source-linked AI summary

APT: Feedback-Driven Agentic Post-Training for Text-to-Image Generation

Sirui Zhang, Fangyun Wei, Dong Chen, Jinjing Zhao, Yitong Wang, Yunuo Chen, Baining Guo, Zheng-Jun Zha

arXiv:2610.18097v1cs.CVcs.AI

TL;DR

Existing post-training often relies on manually curated data and fixed distributions, despite differences in models’ capability profiles. APT automates evaluation, diagnosis, and model-adapted RL data generation in an iterative loop, with a broad evaluation harness for general-purpose use. Experiments report higher average performance than handcrafted RL data across models and support model-specific, iterative adaptation.

  • Problem

    Existing post-training typically uses manually constructed training distributions, which are labor-intensive to scale and may not suit models with different capability profiles.

    Source: §1 · p. 2

  • Method

    APT coordinates evaluation-set generation, model diagnosis, and adaptive RL data generation in an iterative loop that tailors training to the current model.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    APT achieves higher average performance than handcrafted RL training data across four base models, while transfer studies and ablations support model-specific data and iterative feedback.

    Source: §3.2 · p. 7, §3.3 · p. 8, §4 · p. 9

  • Takeaways & Limitations

    Feedback-driven task adaptation is a practical direction for automated text-to-image post-training and a step toward recursive self-improvement.

    Source: §4 · p. 9

    APT characterizes capability distributions empirically using broad evaluation coverage and fine-grained, prompt-specific rubric feedback.

    Source: §2.3 · p. 5

Abstract

from arXiv · show

Post-training is crucial for improving text-to-image models, yet existing pipelines typically rely on manually curated training data and fixed training distributions. We introduce APT (Agentic Post-Training), a fully automated framework that adaptively post-trains a base model according to its evolving capability profile. APT coordinates three agents to construct evaluation sets from user requirements, diagnose model-specific strengths and failure modes, and generate reinforcement learning data with an adaptive prompt distribution that emphasizes underperforming capabilities. By repeatedly evaluating, diagnosing, and updating the generator, APT lets the current model's behavior shape its next training tasks, forming a feedback loop inspired by recursive self-improvement. For general-purpose settings without user-specified evaluation scenarios, we further develop an evaluation harness that enables broad and diverse evaluation set generation. Experiments across multiple base models, benchmarks, and generation architectures demonstrate that APT is model-agnostic, benchmark-agnostic, architecture-agnostic, and fully automated.

1 INTRODUCTION

Existing post-training often depends on labor-intensive, manually adjusted training distributions, while different models have different capability and failure profiles. APT automates an evaluation-driven loop that adapts training data to the current model, including a broad evaluation harness for general-purpose use; experiments report consistent gains across diverse models and settings.

1 INTRODUCTION

  • Existing post-training typically relies on manually constructed training distributions, a labor-intensive workflow that is difficult to scale as models, benchmarks, and user requirements diversify.Different base models can have distinct capability and failure distributions, so data effective for one model is not necessarily optimal for another.Source: §1 · p. 2
  • APT evaluates the current model, diagnoses strengths and failure modes, and generates model-specific RL data that reshapes the next round’s training distribution.Its three agents construct evaluation suites, diagnose at instance and category levels, and prioritize underperforming capabilities while retaining moderate and mastered ones.Source: §1 · p. 2
  • When users provide no evaluation scope, APT uses a harness to construct a broad, diverse evaluation set spanning common generation scenarios and capability dimensions.Source: §1 · p. 2
  • APT consistently improves diverse base models and outperforms carefully handcrafted RL training data, while results support model-specific, evolving training distributions.The framework is validated across multiple models, generation architectures, and benchmarks.Source: §1 · p. 2

2 METHOD

APT uses repeated evaluation, diagnosis, and reinforcement-learning updates to adapt training tasks to a text-to-image model’s evolving weaknesses. Its agents construct evaluation sets, produce model-specific training prompts, and support application across models, benchmarks, and architectures.

2 METHOD

  • At each round, APT diagnoses the current model, generates training data from the report, applies RL updates, and reevaluates the updated model to guide the next round.The evaluation set is fixed after its initial construction, while the training distribution adapts to evolving weaknesses.Source: §2 · p. 3

2.1 EVALUATION SET GENERATION AGENT

  • The evaluation-set agent builds a hierarchical scenario taxonomy, combines items with compositional descriptions, and generates prompt-specific rubrics, producing 2,609 prompt–rubrics pairs.Figure 2 presents category-level statistics and an example prompt–rubrics pair.Source: §2.1 · p. 4, Figure 2 · p. 4

2.2 DIAGNOSIS AGENT

  • The Diagnosis Agent samples prompt–rubric pairs and records rubric-level results in a report that captures the current model’s successes and failures across generation capabilities.The diagnostic batch size is 120 by default.Source: §2.2 · pp. 4–5

2.3 RL DATA GENERATION AGENT

  • The RL Data Generation Agent prioritizes recurring weak capabilities while retaining prompts for moderately performing and strong capabilities, updating the model-specific training distribution over rounds.Its report-based capability estimates use broad evaluation coverage, prompt-specific rubrics, and cross-example pattern summarization.Source: §2.3 · p. 5

2.4 GENERAL, ADAPTIVE, AND FULLY AUTOMATED POST-TRAINING

  • APT is described as benchmark-agnostic, model-agnostic, architecture-agnostic, and fully automated, without requiring task- or model-specific redesign.The same workflow is described for diverse base models and both diffusion-based and autoregressive generation models.Source: §2.4 · p. 6
  • APT operationalizes recursive self-improvement by using each model update to change its capability profile and thereby revise subsequent training priorities.The paper describes this as replacing manual intervention in deciding what the model should learn next.Source: §2.4 · p. 6

3 EXPERIMENTS

APT is evaluated across diffusion and autoregressive models, five benchmarks, and multiple ablations. Results favor adaptive, model-specific training and iterative feedback over the tested alternatives.

3.1 EXPERIMENTAL SETUP

  • The experiments cover four diffusion models and Janus-Pro, and evaluate compositional generation, prompt following, and text rendering across five benchmarks.None of the five benchmarks is used to construct training prompts.Source: §3.1 · p. 6

3.2 MAIN RESULTS

  • APT achieves the best average performance across four base models and improves multiple benchmarks with one post-training run, outperforming the handcrafted RL baseline.The same agentic workflow is used across models without model-specific prompt sets, reward designs, or hyperparameter tuning.Source: Table 1 · p. 7, §3.2 · p. 7

3.3 ABLATION STUDY

  • For every target model, adaptively generated RL data achieves the best average performance within its row, while data generated for another model performs worse.The transfer comparison post-trains each model for 200 steps.Source: §3.3 · p. 8, Table 2 · p. 8
  • APT-generated evaluation sets yield the best average performance on Lens and SD3.5-Medium; single-benchmark evaluation sets bias diagnosis toward their corresponding capability.The comparison replaces APT’s evaluation set while keeping the rest of the pipeline and training configuration unchanged.Source: §3.3 · p. 8, Table 3 · p. 8
  • Removing re-diagnosis after each training round consistently degrades performance, despite keeping the number of prompts and RL optimization steps unchanged.The ablation tests continued prompt generation without feedback from the updated model.Source: §3.3 · p. 9, Table 4 · p. 9
  • Removing diagnostic criteria or sampling only the hardest capabilities is inferior to the full RL prompt-sampling strategy.The full strategy prioritizes weak capabilities while retaining moderate and mastered ones.Source: §3.3 · p. 9, Table 4 · p. 9
  • On Janus-Pro, APT outperforms both the original model and handcrafted RL baseline across evaluated benchmarks, extending results to autoregressive generation.The evaluation uses GRPO for RL post-training.Source: §3.3 · p. 9, Table 5 · p. 9
  • All tested LLM backbones yield substantial Lens average-performance gains, indicating robustness across the evaluated backbone families and scales.Absolute performance differs among the tested backbones.Source: §3.3 · p. 9

4 CONCLUSION

APT adapts text-to-image post-training to each model’s evolving capabilities by coordinating evaluation, diagnosis, and RL data generation. Experiments report higher average performance than handcrafted RL data, while transfer studies and ablations support model-specific training and iterative feedback.

4 CONCLUSION

  • APT is a fully automated framework for adaptive post-training of text-to-image models.It coordinates evaluation set generation, model diagnosis, and RL data generation to adapt training to evolving capabilities.Source: §4 · p. 9
  • Across multiple models and benchmarks, APT achieves higher average performance than post-training on handcrafted RL data.Source: §4 · p. 9
  • Transfer studies and ablations support model-specific training data and iterative feedback as practical directions for automated post-training.The conclusion describes this as a step toward recursive self-improvement in text-to-image generation.Source: §4 · p. 9

A RELATED WORK

Prior T2I post-training includes supervised fine-tuning and reward-based optimization, while agentic generation work often acts around a frozen model at inference time. APT instead places agents in the post-training loop to change the model itself.

A RELATED WORK

  • Supervised fine-tuning adapts T2I models to curated image–text pairs, but depends on the collected corpus and costly human curation.The passage describes data filtering, recaptioning, and aesthetic curation as important to its performance gains.Source: A RELATED WORK · p. 13
  • Reward-based approaches include policy-gradient optimization, diffusion-adapted preference optimization, step-aware preference signals, and online policy gradients for flow matching.The passage names DDPO, Diffusion-DPO, SPO, Flow-GRPO, and DiffusionNFT among these methods.Source: A RELATED WORK · p. 13
  • Prior agentic image-generation methods improve use of a frozen generator at inference time, whereas APT trains the model and retains gains under standard single-pass inference.The passage presents the two directions as complementary: one improves use of a given model; the other improves the model itself.Source: A RELATED WORK · p. 13

B MORE ABLATION STUDIES

APT’s RL Data Generation Agent remains effective with the tested LLM backbones, although their absolute performance differs.

B MORE ABLATION STUDIES

  • All tested LLM backbones achieve comparable improvements across five benchmarks, despite noticeable differences in absolute performance.Source: B MORE ABLATION STUDIES · p. 13
  • The tested backbones span different LLM families and model scales, with other post-training components held fixed.Source: B MORE ABLATION STUDIES · p. 13
  • The results suggest APT’s effectiveness does not depend on a specific LLM family or scale.The authors characterize the RL Data Generation Agent as robust across evaluated proprietary and open-source backbones.Source: B MORE ABLATION STUDIES · p. 13

C DETAILED BENCHMARK RESULTS

Detailed benchmark results compare APT with base models and handcrafted RL across four text-to-image models and multiple benchmarks. On OneIG-EN, APT generally improves capability-oriented and overall scores over handcrafted RL, while RL post-training can reduce diversity.

C DETAILED BENCHMARK RESULTS

  • A separate ablation replaces GPT-5.4 with LLM backbones from different families and scales while keeping the rest of APT unchanged, using Lens across five benchmarks.The supplied table caption describes the setup but does not state the results.Source: Table 6 · p. 14
  • GenEval2 uses Soft-TIFAGM, which aggregates atom-level correctness with a geometric mean within prompts and an arithmetic mean across images.The passage reports this variant because Kamath et al. (2025) found higher AUROC against prompt-level human judgments than Soft-TIFAAM.Source: C DETAILED BENCHMARK RESULTS · p. 14
  • The benchmark descriptions cover object-centric compositional alignment, dense prompt following, and complex visual text generation.GenEval uses six templated tasks; DPG uses dependency-aware question graphs, while CVTG-2K varies text-region count and attributes.Source: C DETAILED BENCHMARK RESULTS · p. 14
  • On OneIG-EN, APT generally achieves higher alignment, text, reasoning, and overall scores than handcrafted RL, despite diversity decreases after RL post-training.The diversity decline can offset gains in other dimensions and occasionally leave an average score below the base model.Source: C DETAILED BENCHMARK RESULTS · p. 14, Table 8 · p. 16
  • Detailed tables compare APT, original base models, and handcrafted RL across four models on GenEval, OneIG-EN, DPG, and CVTG-2K.The supplied table captions identify these benchmark comparisons but do not provide their cell values.Source: Table 7 · p. 15, Table 8 · p. 16, Table 9 · p. 16
  • Additional tables report transfer-data comparisons and ablations of evaluation-set generation, the iterative feedback loop, and RL data generation.The captions identify detailed results on GenEval, OneIG-EN, DPG, and CVTG-2K without supplying the comparisons' values.Source: Table 11 · p. 17, Table 12 · p. 18, Table 13 · p. 18

D VISUALIZATION

The visualizations compare the base model, handcrafted RL baseline, and APT on Lens, SD3.5-Medium, FLUX.1[Dev], and LongCat-Image.

D VISUALIZATION

  • Figures 3–7 provide qualitative comparisons of the base model, handcrafted RL baseline, and APT across four generation models.Figure 3 and Figure 4 show Lens; Figures 5–7 show SD3.5-Medium, FLUX.1[Dev], and LongCat-Image, respectively.Source: Figure 3 · p. 19, Figure 4 · p. 20, Figure 5 · p. 21

E PROMPT

The appendix prompts specify how APT generates evaluation prompts and rubrics, scores images, and turns diagnosed weaknesses into RL training prompts.

E.1 PROMPT FOR EVALUATION PROMPT GENERATION

  • The evaluation-prompt prompt asks for five diverse, vivid image prompts centered on a supplied entity and incorporating every required keypoint.Prompts should vary in setting, lighting, composition, and mood, while remaining concrete and 30–60 words long.Source: §E.1 · p. 24
  • The prompt-generation instructions define keypoints for scene composition, count, position, attributes, color, relations, style, and visible text.Each keypoint must be naturally incorporated into every generated prompt.Source: §E.1 · p. 24
  • The evaluation-prompt and rubric prompts require JSON outputs, while the RL data-generation prompt requires exactly the requested number of JSONL prompt objects.The RL prompt format specifies an integer id and a prompt string for each object.Source: §E.1 · p. 24, §E.2 · p. 25, §E.4 · p. 27

E.2 PROMPT FOR RUBRIC GENERATION

  • The rubric-generation prompt selects evaluation categories only when relevant to the image prompt, including OCR alignment, object count, attributes, actions, and spatial reasoning.It requests up to ten rubrics, each with a short name and brief explanation.Source: §E.2 · p. 25

E.3 PROMPT FOR RL REWARD GENERATION

  • For RL reward generation, the fixed evaluator returns 1 when an image fully satisfies a criterion and 0 when it clearly fails.These binary criterion scores are aggregated into the scalar training reward.Source: §E.3 · pp. 25–26

E.4 PROMPT FOR THE RL DATA GENERATION AGENT

  • The RL data-generation prompt analyzes low-reward prompts and per-criterion failures to identify recurring visual demands the model handles poorly.It emphasizes task characteristics such as object attributes, spatial relations, text rendering, and visual style rather than semantic topic similarity.Source: §E.4 · p. 26
  • Generated RL prompts use a mixed difficulty distribution: most target weak capabilities, with smaller portions for unstable and stronger capabilities.The instructions call for diverse, new, natural, visually specific, and testable prompts, while retaining breadth rather than focusing only on the weakest category.Source: §E.4 · p. 26

E.5 PROMPT FOR THE RL DATA GENERATION AGENT IN THE REDUCED DIAGNOSTIC CRITERIA ABLATION

This reduced-diagnostic-criteria ablation prompt retains reward-guided failure analysis and mixed-difficulty RL prompt generation, with a narrower description of failure patterns.

E.5 PROMPT FOR THE RL DATA GENERATION AGENT IN THE REDUCED DIAGNOSTIC CRITERIA ABLATION

  • The reduced-diagnostic prompt analyzes low-reward prompts and per-criterion failures to infer visual demands the model handles poorly.Its listed patterns focus on object types and visual style or medium.Source: §E.5 · p. 27
  • It requests a mixed training-prompt distribution weighted toward weak capabilities, with smaller portions for medium-strength and stronger capabilities.The prompt discourages focusing only on the single weakest category and asks for diverse, natural, visually specific, testable prompts.Source: §E.5 · p. 27
  • The output instructions require the requested number of JSONL lines, each containing an integer id and a training prompt string.The prompt also directs the agent to return only the JSONL prompt objects.Source: §E.5 · pp. 27–28

E.6 PROMPT FOR THE RL DATA GENERATION AGENT IN THE HARD-ONLY SAMPLING ABLATION

The hard-only sampling ablation prompt directs the agent to infer visual failure patterns from reward-scored evaluations, then generate new training prompts aimed at recurring weaknesses.

E.6 PROMPT FOR THE RL DATA GENERATION AGENT IN THE HARD-ONLY SAMPLING ABLATION

  • The agent analyzes the full reward spectrum with primary emphasis on low-reward prompts, using average rewards and per-criterion failures to identify visual-task weaknesses rather than topic similarity.Source: §E.6 · p. 28
  • The agent must generate new prompts targeting weak or recurring failure modes, prioritizing frequent, severe, and persistent errors while avoiding well-handled capabilities.Source: §E.6 · p. 28
  • Generated prompts must be diverse, natural, realistic, visually specific, and testable, without merely copying evaluation prompts.Source: §E.6 · p. 28
  • The agent must return exactly N JSONL prompt objects with integer IDs and no surrounding explanation or other text.Source: §E.6 · p. 28

Related papers

Loading 2610.18097v1…