PaperlayerView exact revision on arXiv

Source-linked AI summary

UniEvo-VL: An On-policy Self-Distillation Training Recipe for Multimodal Model Self-improvement

Fang Wu, Da Xing, Yanjie Huang, Junxi Wang, Ji Wang, Hejia Geng, Guancheng Wan, Bowen Zuo, Xiaomin Li, Shixiang Tang, Xinyu Xiang, Zehong Wang, Shiyi Du, Peng Xia, Shuangjia Zheng, Yining Hong, Li Erran Li, Jure Leskovec, Yejin Choi

arXiv:2609.38721v1cs.AIcs.CV

TL;DR

Multimodal models can critique their own generations, but prior methods do not directly transfer that corrective information into original-prompt generation along the model's own trajectories. UniEvo-VL trains a student from a critique-conditioned teacher's predictions. Training improves reported generation scores, while reflection remains beneficial and evaluator limitations qualify how scores should be read.

  • Problem

    Prior self-improvement approaches do not directly distill critique-derived corrections into the original-prompt policy along its own sampling trajectories.

    Source: §1 · p. 2

  • Method

    UniEvo-VL uses a critique-derived revised prompt as privileged teacher context and distills the teacher's predictions along the student's original-prompt sampling trajectories.

    Source: §1 · p. 2, §2.3 · p. 4

  • Results

    Training improves native scores with and without reflection on paired GenEval, GenEval2, and OCR evaluations, with training plus reflection performing best; GenEval rises from 0.747 to 0.808.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The findings support critique-conditioned self-distillation as a route to improving unassisted generation alongside feedback-guided generation.

    Source: §5 · p. 10

    GenEval's detection, classification, and geometric-rule errors can penalize prompt-satisfying images, so its score alone does not establish semantic correctness.

    Source: §D.7 · p. 21

Abstract

from arXiv · show

Modern multimodal models bring generation and understanding into a single unified system, which enables them to provide and learn from their own feedback. Motivated by this unified capacity, we introduce UniEvo-VL, a self-evolving framework for multimodal models to learn from this constructive self-correction feedback during test-time compute. Instead of relying on a separate, often larger, teacher, we leverage their self-critiques as privileged information and ask a single multimodal model to act as both teacher and student with different contexts. The student only sees the vanilla question, while the teacher conditions on the privileged critique. Then training minimizes the per-state divergence between their denoising diffusion distributions over the student's own sampling trajectories. Experiments demonstrate that UniEvo-VL improves the image generation capabilities of multimodal models, while maintaining their sensitivity to additional reflection information. Specifically, we build on top of the open-source Qwen-image-2512 and observe a significant performance gain from 0.747 to 0.808 on GenEval and from 32.97 to 35.53 on GenEval2 Soft-TIFA. Moreover, attempts with more powerful external critics (e.g., GPT5.6-Luna) show that multimodal models with strong judge capabilities can anticipate a higher self-evolving ceiling. Last but not least, mixed text-rendering outcomes show that our self-improvements may not be uniform across different tasks. Our study aims to shed light on the current hot recursive self-improvement research line to enhance the user experience when using multimodal models without external supervision or guidance.

1 INTRODUCTION

Prior self-improvement approaches do not directly distill critique-derived corrections into the original-prompt policy along its own sampling trajectories. UniEvo-VL uses privileged critique-conditioned teacher predictions to train that policy, improving reported generation scores.

1 INTRODUCTION

  • Prior self-improvement methods do not directly distill corrective conditioning into the original-prompt policy along its own sampling trajectories.A critique can specify a desired correction without specifying how intermediate denoising predictions should change.Source: §1 · p. 2
  • UniEvo-VL turns critique into a revised privileged prompt for the teacher, while the student learns from teacher predictions using only the original prompt.This on-policy self-distillation uses neither corrected-image targets nor scalar rewards for policy optimization.Source: §1 · pp. 2–3
  • GenEval rises from 0.747 to 0.808, while GenEval2 Soft-TIFA rises from 32.97 to 35.53 with corrective guidance.The paired evaluation also finds that the evolved generator continues to benefit from an additional reflection pass.Source: §1 · p. 2
  • The paper compares direct and reflection-assisted generation before and after training to distinguish retained gains from reflection's additional benefit.Its analyses also investigate critic choice and post-revision verification across compositional generation and text rendering.Source: §1 · p. 3

2 METHOD

UniEvo-VL uses multimodal understanding to generate corrective prompts, then distills a privileged teacher's predictions into a student on the student's own denoising trajectory. The student is trained to generate from the original prompt alone.

2.1 SELF-EVOLUTION ITERATION BETWEEN GENERATION AND UNDERSTANDING

  • The student generates from the vanilla prompt, while an EMA teacher can condition on the critique-derived privileged modification.A discrepancy and binary acceptance assessment identify outputs needing revision; unusable critic responses are removed.Source: §2.1 · pp. 3–4

2.2 CORRECTIVE CONDITIONING AND EXPERIENCE ACQUISITION

  • For a failed draft, the model synthesizes a revised prompt that preserves the requested scene while addressing identified discrepancies.A regenerated image under the revised prompt is assessed against the original prompt; only a valid accepted result qualifies the revised prompt for training, and the image itself is not a training target.Source: §2.2 · p. 4

2.3 ON-POLICY SELF DISTILLATION

  • On-policy self-distillation matches teacher and student denoising transitions at states visited by the student, transferring privileged corrections into original-prompt generation.The student trajectory starts from the vanilla prompt; the flow-based implementation uses squared distance between deterministic latent transitions.Source: §2.3 · pp. 4–5

3 EXPERIMENTS

Across three image-generation tasks, UniEvo-VL with revision verification exceeds Base on every listed metric, while metric leaders vary by task; the experiments also examine prompt difficulty, reflection, and critic choice.

3.1 EXPERIMENTAL SETUP

  • Experiments update only generator LoRA parameters separately for GenEval, GenEval2, and OCR, and evaluate direct and one-critique generation against original prompts without using evaluation scores as training rewards.The setup uses Qwen-Image-2512 with a fixed Qwen-VL feedback pipeline.Source: §3.1 · p. 5

3.2 UNIFIED DIRECT-GENERATION RESULTS

  • The revision-verified Qwen configuration exceeds Base on every listed metric across all three tasks, although different configurations lead on individual metrics.Table 1 compares the reported configurations after OPSD.Source: §3.2 · p. 5, Table 1 · p. 6
  • Compositional metric leaders differ, while verified Qwen leads OCR; unverified Qwen and Luna have mixed OCR outcomes across native text fidelity and HumanPref.Luna leads native score and atomic accuracy on GenEval and GenEval2, whereas verified Qwen leads OCR on native score and HumanPref.Source: §3.2 · pp. 6–7

3.3 IMPROVEMENT AND PRESERVATION ACROSS PROMPT DIFFICULTY

  • On baseline-defined 500-prompt groups, gains concentrate on Hard prompts while Easy averages fall 1–4 normalized points; acquisition favors drafts needing revision, but grouping does not isolate filtering.The Hard/Easy groups remain fixed after training, and their different improvement headroom prevents attributing the contrast to filtering alone.Source: Figure 4 · p. 7, §3.3 · p. 7

3.4.1 DOES TRAINING TRANSFER THE BENEFIT OF REFLECTION?

  • Direct generation improves on every reported metric, reflection remains beneficial after training, and under matched reflection UniEvo-VL exceeds the reflected base on all three native scores.The native GenEval reflection gain falls from 0.078 to 0.030, while reflection gains remain substantial on GenEval2 and change little on OCR.Source: §3.4.1 · pp. 7–8, Table 2 · p. 8

3.4.2 HOW DOES CRITIC CAPACITY AFFECT SELF-EVOLUTION?

  • GPT-5.6-Luna yields larger GenEval2 gains in two-object composition and color and a higher overall GenEval endpoint than Qwen, while GenEval2 position and attribute gains remain limited.The reported two-object and color gains are approximately +1.21 and +1.02 for Luna, versus +0.28 and +0.37 for Qwen.Source: §3.4.2 · pp. 8–9, Table 3 · p. 9

4 RELATED WORK

Prior work uses multimodal understanding for image scoring, reward-based training, role-based interactions, and iterative refinement. UniEvo-VL instead transfers critique-derived corrections into direct generation and distinguishes learned gains from reflection’s additional benefit.

Understanding and reflection for image generation

  • UniEvo-VL transfers critique-derived corrections into direct generation from the original request, while separately measuring the additional benefit of reflection after training.Earlier approaches include scoring and preference-data construction, self-reward training, role-based interactions, and iterative inference-time refinement.Source: §4 · p. 9

Feedback-driven optimization

  • Feedback-driven methods use rubric rewards, reward-based updates, or detached clean-output targets, whereas UniEvo-VL derives targets from teacher predictions conditioned on corrective text.Its targets require neither gradients through the critic nor a differentiable image reward.Source: §4 · p. 9

On-policy distillation with privileged conditioning

  • Prior on-policy distillation includes teacher predictions, feedback-conditioned language policies, and task-specific diffusion teachers; UniEvo-VL instead derives privileged conditioning from critiques of its own images.The teacher receives a revised prompt and the student the original request, with denoising predictions matched along student trajectories and no paired target images or independently task-trained generator teacher.Source: §4 · pp. 9–10

5 CONCLUSION

The conclusion presents UniEvo-VL as critique-driven on-policy self-distillation, reporting native-score gains with and without reflection alongside variation across configurations and regressions on initially easier prompts. The appendices specify a flow-based transition-matching objective, acquisition filtering, and a fixed-base, EMA-reference setup.

5 CONCLUSION

  • Paired evaluations on GenEval, GenEval2, and OCR show training improves native scores with and without reflection, while combined training and reflection performs best on these metrics.Gains vary across configurations, and initially easier prompts can regress, indicating limits to preservation.Source: §5 · p. 10

A IMPLEMENTATION AND CONFIGURATION

  • Training updates only image-generation LoRA parameters while the base generation weights, text encoder, VAE, and feedback-producing components remain fixed.Qwen-Image-2512 and Qwen-VL are selected for a generation–understanding setting because the generator incorporates a Qwen2.5-VL conditioning encoder.Source: A IMPLEMENTATION AND CONFIGURATION · p. 13

A.1 FLOW-BASED GENERATION OBJECTIVE

  • The flow objective matches student and reference predictions on the same detached student-trajectory states, with the reference conditioned on the revised prompt and the student on the original prompt.The flow discrepancy is squared velocity distance weighted by the squared sampler interval; it is deterministic transition matching, not an exact stochastic-policy KL or JS divergence.Source: §A.1 · p. 13
  • Only selected high-noise transitions receive gradients through student local predictions; rollout collection, assessment, synthesis, and reference targets are detached.For the reported 20-step recipe, six transitions are selected, and the regenerated image in verified OPSD is used only for verification, not as a training target.Source: §A.1 · p. 13
  • Experiments validate only the flow instantiation; an autoregressive extension would require a separately defined loss on next-token distributions at a shared prefix and vocabulary.The general framework requires architecture-compatible local predictions.Source: §A.1 · p. 13

A.2 ACQUISITION AND REFERENCE PARAMETERS

  • Acquisition can admit a revised-prompt example directly or, with post-revision verification, regenerate once and retain it only if the revised image is accepted under the original prompt.The regenerated image filters acquisition but is never the distillation target; during OPSD, the reference uses the revised prompt and the student the original prompt.Source: §A.2 · p. 14
  • The reference generation adapter is updated by EMA after optimizer steps, using decay 0.999 in the main experiments, while base generator weights remain shared and fixed.Reference predictions are computed without gradients.Source: §A.2 · p. 14

A.3 TWO TRAINING RECIPES

The appendix documents training recipes for acquisition with and without post-revision verification and reports an EMA-decay sweep favoring slower updates within the tested range.

A.3 TWO TRAINING RECIPES

  • Tables 4 and 5 specify recipes without and with post-revision verification, respectively, and differ in feedback backend, acquisition seeds, learning rate, and gradient accumulation.Table 4 includes unverified Qwen counterparts on GenEval2 and OCR alongside Luna configurations.Source: Table 4 · p. 14, §A.3 · p. 14, Table 5 · p. 15

A.5 TEACHER PARAMETERIZATION

  • On GenEval, both Task and HumanPref scores favor EMA decays 0.995 and 0.999 over 0.99, with 0.995 achieving the highest scores in this sweep.Source: Figure 5 · p. 15

A.4 TRAINING COMPUTE AND ITERATION COUNT

The appendix reports that slower EMA updates perform better within the tested range, while describing compute overhead from acquisition and the critique interface that converts observed failures into corrective prompts.

A.4 TRAINING COMPUTE AND ITERATION COUNT

  • With post-revision verification, approximately 10–21% of acquisition attempts are accepted, or roughly five to ten attempts per accepted training pair.Rejected candidates add overhead without increasing the update count, and wall-clock cost depends on dataset difficulty, generation cost, and critic latency.Source: §A.4 · p. 15

A.5 TEACHER PARAMETERIZATION

  • The main runs use an EMA teacher with decay 0.999, and within the tested range, slower teacher updates perform better; decreasing EMA further may slightly increase performance but risks instability, as seen with 0.99.The benefit of other schedules, such as a time-varying teacher schedule, remains unexplored.Source: §A.5 · p. 15

B CRITIQUE INTERFACE

  • The critique interface separates semantic, text, and quality checks before synthesis turns actionable observations into a standalone generation prompt.A request with no visible problem can return KEEP.Source: B CRITIQUE INTERFACE · p. 16
  • Prompt synthesis resolves the mismatch between negative descriptions of an output and the coherent desired-output specification needed by the generator.The student lacks the teacher’s additional context, so successful imitation transfers the correction into the original-prompt policy.Source: B CRITIQUE INTERFACE · p. 16
  • Matching teacher and student at the student’s noisy states supplies local guidance on trajectories the student visits and yields a dense signal about how a proposed correction changes its next transition.For deterministic flow sampling, the implementation uses an L2 transition objective rather than requiring a stochastic-policy KL interpretation.Source: B CRITIQUE INTERFACE · p. 16

C DATASETS

The study evaluates compositional image generation with GenEval and GenEval2, and visual text rendering with an OCR prompt collection. These benchmarks use textual specifications, with GenEval2 testing increasingly complex combinations of requirements.

GenEval: compositional generation

  • GenEval tests object-level requirements across six categories, including object count, color, position, and attribute binding.Its standard set contains 553 prompts; training prompts come from GenEval-format training metadata.Source: C DATASETS · p. 16

GenEval2: compositional generation

  • GenEval2 evaluates harder prompts combining 3–10 semantic requirements, using 800 prompts with fixed prompt–seed assignments for holistic evaluation.Its separate training stream uses a compositional training collection.Source: C DATASETS · pp. 16–17

OCR: visual text rendering

  • The OCR task tests whether generated images reproduce a quoted target string readably within a described scene.The collection has approximately 20,000 training prompts and 1,018 test prompts.Source: C DATASETS · p. 17

D EVALUATION DEFINITIONS AND ADDITIONAL RESULTS

Evaluation separates direct generation from generation with corrective feedback and assesses both against the original prompt. The full-cohort verified recipe uses specified image counts, while prompts and seeds are paired within experiments.

Evaluation protocol

  • Direct generation and generation with one corrective-feedback opportunity are reported separately, and both are judged against the original benchmark prompt.The revised generation prompt does not replace the evaluation target.Source: D EVALUATION DEFINITIONS AND… · p. 17
  • Prompts and generation seeds are paired across checkpoints and inference conditions within each experiment, not across verified and unverified configurations.The generation–reflection/model-gap study uses 553 unique GenEval prompts; its settings are not pooled with full-cohort evaluations.Source: D EVALUATION DEFINITIONS AND… · p. 17

Evaluation cohorts

  • Full-cohort holistic learning curves use 553 GenEval, 800 GenEval2, and 1,018 OCR prompts, with one image per prompt.The full-cohort recipe uses 1024 × 1024 images, 20 denoising steps, and guidance scale 4.0.Source: D EVALUATION DEFINITIONS AND… · p. 17

Full-cohort evaluation

  • The full-cohort verified recipe reports Gemini evaluations with image counts specified per checkpoint and inference condition.HumanPref is evaluated separately on each corresponding image pool.Source: Table 6 · p. 17

D.1 GEMINI EVALUATION

The evaluation appendix specifies Gemini scoring procedures and distinguishes those judgments from native benchmark metrics. It also cautions that each score reflects its evaluator and aggregation rule, rather than a single interchangeable measure.

D.1 GEMINI EVALUATION

  • Gemini judging uses gemini-2.5-flash at temperature 0 with judge seed 42, schema-constrained responses, and retry handling for service failures.Unrecoverable full-holistic requests stop scoring rather than yielding a partially complete cohort.Source: §D.1 · pp. 17–18
  • Atom-based checks come from benchmark annotations; check accuracy weights checks equally, while strict success requires every check for an image to pass.Full GenEval2 evaluation uses 6,012 checks and 800 images; verified GenEval strict success averages over 2,212 images.Source: §D.1 · p. 18
  • Holistic Gemini task ratings score compositional fidelity or rendered-text fidelity on a 0–10 scale, separately from atom accuracy.HumanPref is an automated absolute proxy, not a human annotation study or pairwise win rate, and is not combined with semantic correctness.Source: §D.1 · p. 18

D.2 NATIVE BENCHMARK METRICS

  • Official GenEval averages six category accuracies using its detector-based evaluator, while GenEval2 Soft-TIFA summarizes VQA answer probabilities with geometric and arithmetic means.Flow-GRPO OCR uses PaddleOCR text recognition and normalized edit-distance scoring, distinct from Gemini OCR ratings.Source: §D.2 · pp. 18–19
  • Detector errors can penalize prompt-satisfying images, and Gemini judgments remain automated, so each metric should be interpreted by its own evaluator and aggregation rule.Reward-free training does not optimize these scores directly.Source: §D.2 · p. 19

D.3 COMPARISON WITH OTHER METHODS

The comparisons report gains on GenEval, describe the role of post-revision verification, and examine difficulty-specific and GenEval2 trajectories. Historical difficulty results and later trajectory points have narrower scopes than the main comparison.

D.3 COMPARISON WITH OTHER METHODS

  • 0.81 direct and 0.82 with verification compare with the reported Qwen-Image-2512 baseline of 0.75; one critique iteration raises both configurations to 0.85.The verified configuration matches BAGEL’s reported overall score in direct generation and reaches the highest spatial-position score, 0.81, with one critique iteration.Source: Table 7 · p. 19, §D.3 · pp. 19–20

D.4 FULL JUDGE-LESS LEARNING CURVE

  • Post-revision verification checks whether a revised image resolves the original error before accepting the revision for training.The authors present it as a safeguard because prompt revision alone does not guarantee correction, and unverified trajectories fluctuate on GenEval2 and OCR.Source: §D.4 · p. 20

D.5 HISTORICAL DIFFICULTY ANALYSIS ON 500-PROMPT SUBSETS

  • Historical 500-prompt subsets define Hard and Easy groups from baseline scores; their membership stays fixed, and the small OCR Hard subset warrants caution.Hard contains 51, 144, and 24 prompts across the three tasks, respectively; reported normalized changes are percentage-point changes on a 0–100 score.Source: §D.5 · p. 20

D.6 NATIVE GENEVAL2 SOFT-TIFA ACROSS UPDATES

  • Update 200 is the main GenEval2 result across 800 prompts, rather than the later update-220 peak; Figure 6 shows the full 0–300 trajectory.The figure marks the Qwen-critic baseline and update-200 endpoints; GM and AM use each run’s own baseline.Source: §D.6 · p. 20, Table 9 · p. 21, Figure 6 · p. 21

D.7 GENEVAL EVALUATOR LIMITATIONS

This section qualifies how to interpret evaluation and appearance outcomes, records unsuccessful SFT attempts, and documents the critique-to-revision interface. It describes how the rendered-text critic reports visible issues, how notes are synthesized into an image-generation prompt, and an example correction of an incorrect year in rendered text.

D.7 GENEVAL EVALUATOR LIMITATIONS

  • 96.7% human-assessed correctness for Gemini 2.5 Flash Image with prompt rewriting contrasts with 82.1% automatic GenEval, illustrating that its rules can penalize prompt-satisfying images.GenEval is used only for evaluation, not training; a score below 0.95 should not automatically imply a semantic deficit, so progress is also assessed with GenEval2 and qualitative inspection.Source: §D.7 · p. 21

D.8 UNRETURNED JUDGE RESPONSES

  • Ten Gemini task judgments and visual-preference ratings went unreturned after API block responses, which do not establish prompt violations or justify treating them as negative judgments.The diagnostic requests returned no candidate or parsable answer, and the API gave block_reason=OTHER without safety-category ratings.Source: §D.8 · p. 21

D.9 IMAGE COMPARISON

  • Some GenEval and GenEval2 outputs have muted colors and plain, light backgrounds, consistent with realism feedback-prompt bias despite higher scores on selected human-preference benchmarks.The pattern motivates caution because small aesthetic-prompt biases may accumulate during self-evolution.Source: §D.9 · p. 21

E PRELIMINARY ATTEMPTS AT SUPERVISED FINE-TUNING

  • SFT on accepted critique-refined images did not improve generation and rapidly degraded the student; OPSD was easier to make effective.The SFT recipe used an accepted refined image as the training target for its original prompt.Source: E PRELIMINARY ATTEMPTS AT SUPERVISED… · pp. 22–23

F.1 CRITIC AND MERGER INTERFACE

  • The critic separately checks semantic fidelity, rendered text, and visual quality; synthesis combines the original prompt with actionable notes into a revision or emits KEEP, and only verified revisions are retained.The revised text, not the regenerated image, provides the privileged teacher condition during distillation.Source: §F.1 · pp. 23–25

F.4 EXACT RENDERED-TEXT CRITIC PROMPT

  • The rendered-text critic must ground its verdict in visible evidence and return either exactly “No issue on this lens” or one note specifying the issue and complete end-state fix.The prompt prohibits invented flaws and requires a verdict-only response without preface or deliberation.Source: §F.4 · p. 28

F.5 EXACT PROMPT-SYNTHESIS REQUEST

  • The synthesis prompt requires a self-contained revision that preserves the original request and every specific correction without inventing details, or returns KEEP when no real issue remains.It also says correctness outranks polish and instructs the synthesizer to preserve hard attributes, object relations, and required text.Source: §F.5 · p. 30

F.7 EXAMPLE 2: CORRECTING REQUIRED TEXT

  • The rendered year was incorrect: “Freedom 1777” instead of “Freedom 1776”; the fix specifies displaying “Freedom 1776” prominently and legibly above the central arch, nowhere else.The quality lens supplied no actionable note.Source: §F.7 · p. 31

F.6 EXAMPLE 1: REPAIRING AN OBJECT-COUNT ERROR

The example shows a generated two-sink image being critiqued for an object-count error, followed by a corrective revised prompt. The quality and text lenses provide no actionable notes.

F.6 EXAMPLE 1: REPAIRING AN OBJECT-COUNT ERROR

  • The semantic critique identifies that one sink is present when two are required and specifies the count correction.It also calls for white basins, silver faucets, pedestal bases, side-by-side placement, and no overlap or fusion.Source: §F.6 · p. 30
  • The quality and text lenses supply no actionable note.Source: §F.6 · p. 30
  • The revised prompt requests two sinks with the specified attributes and side-by-side placement against the tiled wall.Source: §F.6 · p. 31

Related papers

Loading 2609.38721v1…