PaperlayerView exact revision on arXiv

Source-linked AI summary

RAISE: Requirement-Adaptive Evolutionary Refinement for Training-Free Text-to-Image Alignment

Liyao Jiang, Ruichen Chen, Chao Gao, Di Niu

arXiv:2603.00483v1cs.CVcs.AI

TL;DR

Complex prompts make faithful T2I alignment difficult, while existing scaling methods may use fixed computation or require costly reflection training. RAISE introduces training-free requirement-adaptive evolution with multi-action refinement and checklist-based verification, achieving strong alignment with fewer samples and VLM calls.

  • Problem

    Faithful T2I alignment remains challenging for complex prompts with multiple objects, relations, and fine-grained attributes, while existing methods require fixed budgets or costly reflection training.

  • Method

    RAISE evolves candidate images through prompt rewriting, noise resampling, and instructional editing, verifying requirements with structured tool-grounded checks and allocating computation only when needed.

  • Results

    RAISE achieves 0.94 overall GenEval and 0.885 VQAScore while using 30−40% fewer generated samples and 80% fewer VLM calls than training-based reflection-tuned baselines.

  • Takeaways & Limitations

    RAISE provides adaptive, multi-round T2I refinement without additional model training across GenEval and DrawBench evaluations.

Abstract

from arXiv · show

Recent text-to-image (T2I) diffusion models achieve remarkable realism, yet faithful prompt-image alignment remains challenging, particularly for complex prompts with multiple objects, relations, and fine-grained attributes. Existing training-free inference-time scaling methods rely on fixed iteration budgets that cannot adapt to prompt difficulty, while reflection-tuned models require carefully curated reflection datasets and extensive joint fine-tuning of diffusion and vision-language models, often overfitting to reflection paths data and lacking transferability across models. We introduce RAISE (Requirement-Adaptive Self-Improving Evolution), a training-free, requirement-driven evolutionary framework for adaptive T2I generation. RAISE formulates image generation as a requirement-driven adaptive scaling process, evolving a population of candidates at inference time through a diverse set of refinement actions-including prompt rewriting, noise resampling, and instructional editing. Each generation is verified against a structured checklist of requirements, enabling the system to dynamically identify unsatisfied items and allocate further computation only where needed. This achieves adaptive test-time scaling that aligns computational effort with semantic query complexity. On GenEval and DrawBench, RAISE attains state-of-the-art alignment (0.94 overall GenEval) while incurring fewer generated samples (reduced by 30-40%) and VLM calls (reduced by 80%) than prior scaling and reflection-tuned baselines, demonstrating efficient, generalizable, and model-agnostic multi-round self-improvement. Code is available at https://github.com/LiyaoJiang1998/RAISE.

1. Introduction

RAISE addresses difficult prompt–image alignment without retraining by adapting inference-time computation to unsatisfied requirements and combining multiple refinement actions with structured verification.

  • Motivation: Prompt–image alignment remains difficult for complex prompts containing multiple objects, compositional relations, attribute bindings, and spatial arrangements.Large curated multimodal datasets can improve alignment but are described as costly, datainefficient, and difficult to scale to arbitrary prompts.
  • Contribution: RAISE is a training-free, requirement-driven evolutionary framework for improving T2I alignment at inference time.It dynamically allocates computation according to prompt complexity and unsatisfied requirements, enabling multi-round self-improvement without model retraining.
  • Contribution: The framework analyzes unsatisfied requirements such as object presence, attributes, and spatial relations before allocating additional computation.Computation is directed only where needed and can converge once major requirements are satisfied.
  • Contribution: RAISE concurrently explores prompt rewriting, noise resampling, and instructional editing to refine candidate generations.These complementary actions expand the search space and support progressive, self-correcting evolution.
  • Results: 0.94 overall GenEval and 0.885 VQAScore establish RAISE’s reported state-of-the-art alignment on GenEval and DrawBench.The method also requires 30−40% fewer generated samples and 80% fewer VLM calls than training-based reflection-tuned baselines.

2. Related Work

Related work improves T2I alignment through inference-time computation, prompt refinement, image correction, or learned reflection, but each approach uses a distinct refinement strategy and training requirement.

  • Inference-time scaling: Inference-time scaling adds computation during generation without retraining, including noise resampling guided by scoring functions.Noise-level scaling searches for better initial latent samples, but noise sampling alone can yield limited alignment improvements.
  • Training-free methods: TIR sequentially rewrites prompts with a VLM, whereas T2I-Copilot combines noise scaling, prompt refinement, and image editing in a training-free agentic loop.TIR focuses on linguistic refinement, while T2I-Copilot also applies image-level correction.
  • Training-based methods: Reflection-tuned methods jointly fine-tune diffusion and vision-language models to condition later generations on prior images and textual feedback.Reflect-DiT adds a context transformer and fine-tunes the accompanying VLM to produce reflection instructions.

3. Method

RAISE performs adaptive evolutionary T2I refinement by analyzing requirements, applying diverse mutations, and verifying candidates across rounds. It allocates additional computation to unsatisfied requirements and can stop when requirements are sufficiently satisfied or limits are reached.

  • Framework overview: RAISE evolves T2I candidates through requirement analysis, multi-action refinement, and structured tool-grounded verification.Its cooperative agents analyze prompts and feedback, refine prompts or images, and verify generated candidates.
  • Requirement analysis: The analyzer converts prompt semantics into a structured checklist of verifiable conditions, including objects, attributes, spatial relations, and composition.Each requirement is paired with a binary verification question, while analyzer and verifier decisions indicate whether refinement continues.
  • Adaptive scaling: RAISE allocates further refinement rounds to unmet requirements and stops when major or all requirements are satisfied, subject to minimum and maximum round limits.The final output is selected from the best candidate when analyzer or verifier criteria are met, or after K_max rounds.
  • Refinement actions: Its concurrent refinement actions combine noise resampling, prompt rewriting, and instructional editing to explore visual and semantic correction paths.Resampling varies layouts without changing prompt semantics; rewriting addresses unmet requirements; editing applies textual instructions to the best image.
  • Refinement actions: Editing mutations target the most critical requirement, a random unmet requirement, or all unmet requirements, while early rounds emphasize exploration and later rounds add targeted editing.Each mutation produces a candidate from the current global-best parent, using generation or editing models depending on whether a reference image is present.
  • Structured verification: Vision tools provide structured grounding evidence about entities, attributes, and spatial relations for interpretable binary verification by the verifier.The evidence includes scene captions, region labels, bounding boxes, and mean depth values, which support requirement-level decisions.

4. Experiments

RAISE achieves strong prompt–image alignment on GenEval and DrawBench while adapting computation to prompt difficulty and reducing inference costs.

  • Prompt–Image Alignment: 0.94 overall GenEval score surpasses training-free scaling, reflection-tuned methods, diffusion models, and unified multimodal models.RAISE also reaches 0.885 VQAScore on DrawBench.
  • Efficient Adaptive Scaling: 41.9% fewer samples and 88.6% fewer VLM calls are required than training-based inference-time scaling approaches on GenEval.RAISE averages 18.6 samples and 7.3 VLM calls versus 32 and 64, respectively.
  • Efficient Adaptive Scaling: RAISE allocates more computation to semantically complex prompts and converges early on easier ones.It averages 18.6 samples on GenEval and 21.2 on the more reasoning-intensive DrawBench.
  • Reasoning-Intensive Prompts: 33.8% fewer samples and 86.6% fewer VLM calls accompany RAISE’s higher DrawBench VQAScore of 0.885 versus 0.844.Perceptual quality remains stable as alignment improves with additional samples.
  • Visual Comparisons: Requirement-driven evolution improves challenging cases involving implicit requirements, counting, rare attributes, spatial composition, and text-in-image generation.The framework combines rewrite, resample, and edit mutations with structured tool-grounded verification.
  • Ablation Studies: Removing vision tools lowers Attribute Binding and Colors performance, while the full framework achieves the highest overall GenEval score of 0.94.The ablation supports the contribution of structured visual evidence and multi-action refinement.

5. Conclusion

RAISE is a training-free, requirement-driven evolutionary framework for T2I alignment that outperforms training-based inference-time scaling with 40% fewer generated samples. Its gains derive from multi-round refinement, offering an approach for challenging prompts that require repeated refinement.

  • RAISE is a training-free, requirement-driven evolutionary framework that outperforms training-based inference-time scaling with 40% fewer generated samples.
  • RAISE’s observed gains derive from its multi-round refinement capability.
  • RAISE targets challenging and nuanced prompts that mandate multi-round refinement.

Supplementary Material

The supplementary material reports evaluations across different base diffusion and vision-language models, with tables defining efficiency measures and model-label conventions.

  • Table 4 evaluates RAISE with FLUX.1-dev, FLUX.1-schnell, and SANA-1.5 4.8B on GenEval.It reports average generated samples and average VLM calls as efficiency measures.
  • Table 5 evaluates RAISE with different base VLMs on GenEval.The table marks frozen open-source models that require no additional fine-tuning.
  • The supplementary tables distinguish fine-tuned open-source models from proprietary models.

6. Additional Results

Additional experiments show that RAISE transfers across diffusion and vision-language backbones, maintains strong performance across budgets, and allocates computation adaptively during multi-round refinement.

  • Different Base Diffusion Models: RAISE consistently improves prompt–image alignment across tested diffusion models, achieving overall GenEval scores of 0.93–0.94.The tested models vary in size, diffusion steps, and generation quality, and require no diffusion-model fine-tuning.
  • Different Base Vision-Language Models: RAISE delivers consistent alignment improvements and overall GenEval scores of 0.92-0.94 across tested VLM backbones.It requires no VLM fine-tuning or proprietary models.
  • Efficiency and Adaptive Scaling: 41.9% fewer samples and 88.6% fewer VLM calls are required at a 32-sample maximum budget than training-based inference-time scaling approaches on GenEval.RAISE generates 18.6 samples and makes 7.3 VLM calls on average, versus 32 and 64.
  • Efficiency and Adaptive Scaling: RAISE achieves the highest GenEval scores across maximum sample budgets of 8, 16, and 32.Performance continues to improve as the sampling budget increases.
  • Efficiency and Adaptive Scaling: RAISE maintains a strong performance–efficiency Pareto frontier while other methods plateau early or fail to improve with additional samples.
  • Efficiency and Adaptive Scaling: RAISE generates 18.6 samples on GenEval and 21.2 on the more complex, reasoning-intensive DrawBench benchmark.This reflects adaptive allocation of computation to prompts with greater semantic complexity.

7. RAISE Framework

RAISE is a requirement-driven evolutionary framework that iteratively generates, verifies, and refines image candidates through multiple mutation actions until requirements are satisfied or the round budget ends.

  • Multi-Action Evolutionary Refinement: The evolutionary search combines prompt rewriting, noise resampling, and instructional editing to improve alignment across rounds.A bear-and-clock example illustrates adaptive multi-action refinement through requirement verification.
  • Algorithm 1: Requirement-Adaptive Self-Improving Evolution Framework (RAISE): Algorithm 1 takes a user prompt and minimum and maximum round limits, and returns a final output image.
  • Algorithm 1: Requirement-Adaptive Self-Improving Evolution Framework (RAISE): RAISE initializes candidate state and iterates through evolutionary refinement rounds up to the maximum round limit.
  • Algorithm 1: Requirement-Adaptive Self-Improving Evolution Framework (RAISE): The framework performs candidate mutation and requirement verification before selecting candidates for continued refinement.Verification uses tools to extract requirement information and evaluate candidate satisfaction.
  • Analyzer Agent: RAISE analyzes explicit and implicit requirements from the original prompt, current images, and verifier outputs.The analyzer identifies main subjects, exact counts, and other prompt requirements.
  • Analyzer Agent: The analyzer determines which requirements are satisfied or unsatisfied without rewriting the prompt.
  • Generation Rewriter Agent: The generation rewriter plans precise prompt adjustments that address unsatisfied requirements while preserving satisfied ones.The adjusted prompt is intended to improve alignment, coherence, and image quality.
  • Editing Rewriter Agent: The editing rewriter converts unsatisfied requirements into atomic image-editing instructions and selects the most critical edit.Satisfied requirements are preserved unless changing them is necessary to resolve an unsatisfied item.
Loading 2603.00483v1…