Source-linked AI summary
Rethinking Prompt Design for Inference-time Scaling in Text-to-Visual Generation
Subin Kim, Sangwoo Mo, Mamshad Nayeem Rizve, Yiran Xu, Difan Liu, Jinwoo Shin, Tobias Hinz
TL;DR
Complex prompts are difficult to align with generated visuals, and visual-only scaling can leave recurring failures unresolved. PRIS revises prompts using shared failure patterns and EFC’s fine-grained assessments; across T2I and T2V, it achieves stronger prompt adherence and scaling behavior than visual-only methods.
Problem
Complex prompts can be difficult to satisfy in a single generation, and prior refinements focused on individual samples miss recurring failures across scaled visuals.
Source: §1 · p. 1Method
PRIS revises prompts based on shared failures across generated visuals, using EFC to assess alignment at the element level.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
The findings indicate that leveraging recurring failure patterns in prompt redesign is important for realizing inference-time scaling benefits.
Source: §5 · p. 9Extending the approach to other modalities and tasks is identified as future work.
Source: §A.7 · p. 19
Abstract
from arXiv · showhide
Achieving precise alignment between user intent and generated visuals remains a central challenge in text-to-visual generation, as a single attempt often fails to produce the desired output. To handle this, prior approaches mainly scale the visual generation process (e.g., increasing sampling steps or seeds), but this quickly leads to a quality plateau. This limitation arises because the prompt, crucial for guiding generation, is kept fixed. To address this, we propose Prompt Redesign for Inference-time Scaling, coined PRIS, a framework that adaptively revises the prompt during inference in response to the scaled visual generations. The core idea of PRIS is to review the generated visuals, identify recurring failure patterns across visuals, and redesign the prompt accordingly before regenerating the visuals with the revised prompt. To provide precise alignment feedback for prompt revision, we introduce a new verifier, element-level factual correction, which evaluates the alignment between prompt attributes and generated visuals at a fine-grained level, achieving more accurate and interpretable assessments than holistic measures. Extensive experiments on both text-to-image and text-to-video benchmarks demonstrate the effectiveness of our approach, including a 15% gain on VBench 2.0. These results highlight that jointly scaling prompts and visuals is key to fully leveraging scaling laws at inference-time. Visualizations are available at the website: https://subin-kim-cv.github.io/PRIS.
1. Introduction
Complex prompts make it difficult to align generated visuals with user intent, while inference-time scaling often leaves prompts fixed and recurring failures uncorrected. PRIS revises prompts from shared failures during visual scaling, and EFC supplies fine-grained alignment feedback.
1. Introduction
- Complex prompts involving compositional image structures or video motion, camera movement, and causal order are difficult to satisfy in a single generation attempt.Source: §1 · p. 1
- Visual-only scaling keeps prompts fixed, limiting its benefit when prompts are ambiguous or incomplete, while prior per-sample refinements miss failures recurring across samples.The paper illustrates the issue with laces appearing in every output for a prompt requesting a shoe without laces.Source: §1 · p. 1
- PRIS jointly scales prompts and visuals, sustaining text-visual alignment improvements as compute increases where fixed-prompt scaling plateaus.The reported pattern holds for given and unseen rewards.Source: §1 · p. 2, Figure 1 · p. 2
- EFC decomposes prompts into semantic elements and checks each against generated visuals, providing fine-grained feedback that improves verification accuracy and interpretability.The paper reports that EFC distinguishes ground-truth visuals from plausible misaligned distractors and outperforms existing verifiers on its new benchmark.Source: §1 · p. 2
2. Related Work
Prior inference-time scaling expands visual search while holding the prompt fixed, and prompt-refinement methods often operate separately from that scaling. This paper frames shared failures across samples as a basis for revising prompts during visual scaling.
Scaling inference-time compute in visual generation
- Inference-time scaling methods expand visual search by increasing decoding compute or generating candidate pools [12, 18, 19], but keep prompts fixed.Best-of-N selects candidates at the final output stage, while Search-over-Paths [12] propagates high-reward trajectories during sampling.Source: §2 · p. 2
Prompt design in text-to-visual generations
- Prompt-refinement systems can require human involvement and often revise individual samples independently of inference-time scaling.Such isolated rewrites react to individual noisy failures rather than recurring patterns across scaled samples.Source: §2 · p. 3
3. Prompt Redesign for Inference-time Scaling
EFC assesses prompt elements at a fine-grained level, and PRIS uses those assessments to revise prompts based on recurring failures across samples. Its loop selects strong samples, targets shared misalignments, and regenerates with revised prompts.
3.1. EFC: Element-level Factual Correction
- EFC decomposes a prompt into atomic semantic elements, including image objects, properties, spatial arrangement, motion, camera movement, scene transitions, and temporal order.It classifies elements as core factual requirements or extra, more subjective or stylistic attributes.Source: §3.1 · p. 3
- EFC compares each prompt element with a generated visual’s caption using natural-language inference, then uses open-ended visual questions to resolve neutral cases.Element labels are entailment, contradiction, or neutral; responses to follow-up questions are assessed in a second inference step.Source: §3.1 · p. 3
- EFC scores visuals by the number of elements labeled entailment, prioritizing core-element accuracy and using extra-element accuracy to break ties.Source: §3.1 · pp. 3–4
3.2. Common-failure-aware Prompt Redesign for Inference-time Scaling
- PRIS uses EFC’s fine-grained alignment assessments to diagnose recurring failures across samples and revise prompts for subsequent generations.Figure 2 depicts the verifier-guided redesign process and its regeneration loop.Source: §3.2 · p. 4, Figure 2 · p. 4
- PRIS selects top-k samples that cover the most elements, identifies common failures below 50% success within that subset, and revises the prompt to reinforce overlooked elements.If no failures meet that criterion, it instead treats the prompt as a target for exploring variations.Source: §3.2 · p. 4
- PRIS regenerates with the revised prompt while reusing top-k noise latents, then verifies and ranks the outputs; the loop can repeat, though main experiments apply it once.The paper reports that one iteration already provides substantial gains.Source: §3.2 · p. 4
4. Experiments
Experiments test PRIS under fixed and increasing compute, its integration with visual-scaling methods, and EFC as a verifier. PRIS improves prompt alignment across text-to-image and text-to-video settings, while EFC achieves the highest verifier accuracy on the evaluated benchmark.
4. Experiments
- The experiments assess prompt redesign under fixed NFE budgets, scaling behavior and visual-scaling integration, verifier sample selection, and PRIS and EFC ablations.The fixed-budget comparison uses Qwen2.5-VL for EFC without additional training and compares PRIS with BoN and standard prompt expansion.Source: §4 · p. 5, §4.1 · p. 5
4.1. Effect of PRIS on Inference-time Alignment and Visual Quality
- PRIS gains +13.88% in Controllability and +15.19% in Creativity for the small and large T2V models, respectively, and also improves Commonsense and Physics.Commonsense improves by +3.46% and Physics by +6.53%; Thermotics is a slight exception attributed to reward-model overfitting to exact numeric values.Source: §4.1 · pp. 5–6, Table 2 · p. 6
4.2. Scaling Behaviors of PRIS
- Beyond BoN’s plateau above 1e3 NFEs, PRIS continues improving; iterative revisions add alignment gains, and integration with DAS or RBF improves alignment on given and unseen rewards.PRIS also improves aesthetics alongside alignment when integrated with RBF, which often sacrifices aesthetics for alignment.Source: §4.2 · pp. 6–7, Table 3 · p. 7, Table 4 · p. 8
4.3. Evaluating the Verification Accuracy of EFC
- EFC achieves the highest accuracy in selecting aligned videos, surpassing VideoAlign and decomposed VQA while providing interpretable reasoning without training.The benchmark pairs 410 prompts with aligned and partially aligned videos, with human annotations determined by majority vote.Source: §4.3 · pp. 7–8, Table 5 · p. 7
4.4. Ablation Study and Analysis
- Common-failure-aware revision outperforms per-sample revision in T2I and T2V; in T2V, per-sample revision performs worse than no revision.Under matched wall-clock time, verifier-guided revision yields larger prompt-adherence gains than spending the time on more generator NFEs, despite EFC’s modest captioning overhead.Source: §4.4 · p. 8, Table 6 · p. 8, Table 7 · p. 8
5. Conclusion
The conclusion presents PRIS as prompt redesign guided by shared failures across generated samples, alongside EFC for fine-grained alignment assessment. It concludes that jointly scaling prompts and visuals yields stronger scaling behavior than visual-only methods.
5. Conclusion
- PRIS revises prompts from recurring failures across generated outputs, rather than relying on visual-only scaling or isolated per-sample updates.The framework uses EFC-guided feedback to adapt prompts to generator failures.Source: §5 · pp. 8–9
- EFC provides fine-grained text-visual alignment assessments, while PRIS uses shared failure patterns to adaptively update prompts.Source: §5 · p. 8
- Across text-to-image and text-to-video settings, the paper reports substantial prompt-adherence gains and stronger scaling behavior from jointly scaling prompts and visuals.The conclusion identifies shared-failure prompt redesign as crucial to fully realizing inference-time scaling benefits.Source: §5 · pp. 8–9
A. Additional Analysis
The additional analysis illustrates recurring prompt misalignments in generated images and videos, and details how EFC verifies prompt elements. EFC uses captions and factual correction to provide fine-grained, interpretable feedback.
A.1. Qualitative Examples of Common Failures
- Figures 7 and 8 illustrate recurring misalignments across multiple images and videos generated from fixed prompts, with EFC identifying common failures.The video examples show the first and last frame of each generated video.Source: Figure 7 · p. 11, §A.1 · p. 11, Figure 8 · p. 12
A.2. Details of Element-level Factual Correction (EFC)
- EFC decomposes prompts into semantic elements, then uses captions and natural language inference to label each element as entailment, neutral, or contradiction.This provides fine-grained, interpretable feedback on whether each prompt part is realized in the visual.Source: §A.2 · p. 13, Figure 9 · p. 13
- For neutral elements, EFC asks open-ended questions about the visual and compares the answers with target elements, avoiding direct yes/no QA.The second NLI step assigns a final entailment or contradiction label, addressing missing or ambiguous caption information.Source: §A.2 · p. 13, Figure 9 · p. 13
A.3. Details on Integration Beyond BoN
The appendix describes PRIS integrations with image and video visual-scaling methods under specified compute budgets. It reports stronger prompt adherence for image integrations and improved average VBench2.0 scores for EvoSearch integrated with PRIS.
A.3. Details on Integration Beyond BoN
- With eight total image samples split into two batches, PRIS revises the prompt after the first four while keeping total function evaluations equivalent.The integrations use SDXL for DAS and Flux.1-schnell for RBF, following their original protocols.Source: §A.3 · p. 14
- PRIS-integrated DAS and RBF achieve better prompt adherence than visual scaling alone, while PRIS mitigates RBF’s prompt-text rendering artifact.The revised prompt guides generation, while the original prompt is used only for the reward signal.Source: §A.3 · p. 14, Figure 10 · p. 14, Figure 11 · p. 15
- EvoSearch integrated with PRIS improves average VBench2.0 scores on more complex prompts, while EvoSearch scores degrade after scaling.Table 8 compares EvoSearch alone with its PRIS integration; the integration uses 60 fewer NFEs than EvoSearch.Source: §A.3 · p. 17, Table 8 · p. 17
- In the qualitative examples, PRIS changes a butterfly’s wing color and depicts a window opening as a person reaches out, unlike EvoSearch.Source: Figure 13 · p. 17
A.4. Detailed Computational Time Analysis
The appendix details verification costs, compares PRIS with ReflectionFlow, and illustrates prompt transfer to another image generator. It also identifies broader applications and prompt-rewriting training as future directions.
A.4. Detailed Computational Time Analysis
- On one H100 GPU, EFC verification takes 41 seconds per image sample and 100 seconds per 81-frame video, roughly the cost of three images and one video, respectively.The experiments allocate fewer NFEs to PRIS than BoN to account for verification overhead; fine-tuning the pretrained MLLM is proposed as a way to improve efficiency.Source: §A.4 · p. 17
A.5. Comparison with ReflectionFlow
- PRIS is training-free and uses 2000 NFEs versus ReflectionFlow’s 3840, yet the paper reports more accurate, semantically aligned results across diverse categories.ReflectionFlow uses a trained reflection model to edit each sample; PRIS revises prompts from common failures across samples.Source: §A.5 · p. 18, Figure 14 · p. 18
A.6. Prompt Transferability and Future Work
- Prompts revised for Flux1.dev transfer to Firefly Image 4 Ultra, with clarified instructions and contextual cues yielding stronger adherence than original prompts.Examples make object presence and absence explicit and replace vague instructions with more precise ones.Source: §A.6 · p. 19, Figure 15 · p. 19
A.7. Future work and limitations.
- The authors propose extending shared-failure prompt revision to other modalities and tasks, and training prompt-rewriting models on paired naive prompts and failure-focused revisions.They suggest such training resources could reduce verification overhead and inference-time costs.Source: §A.7 · p. 19
B.1. Benchmark Category
The authors construct a benchmark for evaluating prompt-adherence verifiers on video candidates with slight misalignments. On this benchmark, EFC achieves the highest accuracy across categories and outperforms the compared verifier approaches.
B.1. Benchmark Category
- The benchmark is designed to test selection of the best-aligned videos from misaligned candidates, a setting existing human-preference datasets do not adequately cover.The authors present it as a testbed for their verifier, ablations, and existing baselines, as well as a resource for future prompt-adherence verification research.Source: §B.1 · p. 20
- EFC achieves the highest accuracy across all prompt categories in the constructed benchmark.It also substantially outperforms a decomposed binary VQA baseline and maintains a significant lead over learned reward models, including VideoAlign.Source: §B.1 · p. 20, Table 9 · p. 20
- The benchmark contains 410 prompts paired with multiple videos, including at least one ground-truth-aligned reference and other slightly misaligned candidates.Prompts cover motion and physics, plus VBench 2.0 categories for dynamic attributes, motion order, and motion rationality; three evaluators label each video, with majority vote determining the final label.Source: §B.1 · p. 20
- The authors suggest extending their verification framework to motion quality, NSFW filtering, and bias detection using task-specific decomposition strategies.These are proposed directions, not evaluations reported in this section.Source: §B.1 · p. 20
C. Experiments Details
The experiments use Wan as the primary video model because it has reasonable prompt adherence while leaving room for improvement. The setup also subsamples similar GenAI-Bench prompts and sets the number of reviewed candidates based on the generation and seed-selection procedure.
C.1. Detailed Setup
- For GenAI-Bench, the authors randomly subsample 20% of similar prompts and set k = N//4.They first generate N//2 samples for review before prompt revision, then use half of those as top-performing seeds.Source: §C.1 · p. 21
C.2. Base Model Selection
- Base-model prompt adherence is measured by embedding similarity between the original prompt and a generated-video caption.Qwen-32B produces captions, and SentenceTransformer intfloat/e5-mistral-7b-instruct computes similarity; Table 10 reports scores across text-to-video models.Source: §C.2 · p. 21, Table 10 · p. 21
- Wan was selected as the primary video model for its reasonable prompt adherence and remaining scope for improvement; LTX and Hunyuan were excluded for low adherence.The exclusion particularly concerns complex VBench 2.0 prompts involving status changes or multiple consecutive events.Source: §C.2 · p. 21, Table 10 · p. 21
D. Additional Qualitative Experimental Results
Additional qualitative comparisons report improved prompt fidelity for PRIS in text-to-image and text-to-video generation. The examples highlight ambiguous or subtle constraints in images and complex actions or temporal changes in videos.
D.1. Text-to-Image Generation
- PRIS produces more faithful text-to-image realizations of ambiguous attributes, numerical specifications, and subtle constraints under the same NFE budget of 2000.The comparison uses top-scoring outputs from original GenAI-Bench prompts and reports improvements in coherence over baselines.Source: §D.1 · p. 22, Figure 16 · p. 22
- Compared with standard prompt expansion, PRIS achieves substantially higher prompt fidelity by adapting revisions to recurring failure modes.The authors contrast this with standard expansion's inability to target or identify the most challenging semantic elements.Source: §D.1 · p. 23, Figure 17 · p. 23
D.2. Text-to-Video Generation
- In text-to-video examples, PRIS better captures key events, dynamic changes, and sequential actions than BoN, which often omits or misorders them.The examples include static-only outputs and sequences where PRIS depicts both an attempted action and its result.Source: §D.2 · p. 24
D.3. More Visualizations
- Additional visual comparisons include text-to-video generations from both Wan2.1-1.3B and Wan2.1-14B.The attached HTML file presents generated images and videos alongside their prompts and baseline comparisons.Source: §D.3 · p. 24, Figure 18 · p. 25