Source-linked AI summary
RAPO++: Cross-Stage Prompt Optimization for Text-to-Video Generation via Data Alignment and Test-Time Scaling
Bingjie Gao, Qianli Ma, Xiaoxue Wu, Shuai Yang, Guanzhou Lan, Haonan Zhao, Jiaxuan Chen, Qingyang Liu, Yu Qiao, Xinyuan Chen, Yaohui Wang, Li Niu
TL;DR
User prompts are often brief, while existing T2V prompt optimization remains largely model-specific. RAPO++ combines training-aligned refinement, feedback-based iteration, and LLM fine-tuning; across five models and five benchmarks, it improves multiple dimensions of video generation.
Problem
User prompts are often brief, and existing T2V prompt optimization lacks generalizable strategies across diverse models and tasks.
Source: §1 · p. 1Method
RAPO++ combines training-data-aligned prompt refinement, iterative feedback-based optimization, and LLM fine-tuning without changing the generative backbone.
Source: §1 · p. 3, §6 · p. 14Results
Across five T2V models and five benchmarks, RAPO++ improves semantic alignment, compositional reasoning, temporal stability, and physical plausibility; on VBench, its total score is 82.65% with LaVie.
Source: §1 · p. 3, §5.3 · p. 9Takeaways & Limitations
RAPO++ improves compositionality, dynamics, and physical realism while leaving the T2V backbone unchanged.
Source: §6 · p. 14RAPO++ still struggles to generate the specified number of entities, partly because SSPO lacks fine-grained, count-aware feedback.
Source: §5.7 · p. 14
Abstract
from arXiv · showhide
Prompt design plays a crucial role in text-to-video (T2V) generation, yet user-provided prompts are often short, unstructured, and misaligned with training data, limiting the generative potential of diffusion-based T2V models. We present \textbf{RAPO++}, a cross-stage prompt optimization framework that unifies training-data--aligned refinement, test-time iterative scaling, and large language model (LLM) fine-tuning to substantially improve T2V generation without modifying the underlying generative backbone. In \textbf{Stage 1}, Retrieval-Augmented Prompt Optimization (RAPO) enriches user prompts with semantically relevant modifiers retrieved from a relation graph and refactors them to match training distributions, enhancing compositionality and multi-object fidelity. \textbf{Stage 2} introduces Sample-Specific Prompt Optimization (SSPO), a closed-loop mechanism that iteratively refines prompts using multi-source feedback -- including semantic alignment, spatial fidelity, temporal coherence, and task-specific signals such as optical flow -- yielding progressively improved video generation quality. \textbf{Stage 3} leverages optimized prompt pairs from SSPO to fine-tune the rewriter LLM, internalizing task-specific optimization patterns and enabling efficient, high-quality prompt generation even before inference. Extensive experiments across five state-of-the-art T2V models and five benchmarks demonstrate that RAPO++ achieves significant gains in semantic alignment, compositional reasoning, temporal stability, and physical plausibility, outperforming existing methods by large margins. Our results highlight RAPO++ as a model-agnostic, cost-efficient, and scalable solution that sets a new standard for prompt optimization in T2V generation. The code is available at https://github.com/Vchitect/RAPO.
1 INTRODUCTION
RAPO++ addresses short, underspecified prompts and the lack of generalizable T2V prompt optimization with a three-stage framework that aligns prompts to training data, iteratively refines them, and fine-tunes a rewriter LLM. Experiments report improvements across models and benchmarks in semantic alignment, compositional reasoning, temporal stability, and physical plausibility.
1 INTRODUCTION
- User prompts are often brief and underspecified, while T2V prompt optimization remains model-specific; RLHF-style generator rollouts are prohibitively costly for video generation.The cited barriers include video’s heavy temporal structure and higher inference cost, which make naive extensions of T2I RLHF impractical without additional designs.Source: §1 · pp. 1–2
- RAPO retrieves semantically relevant modifiers from a relation graph, refactors enriched prompts toward training-data structure, and uses a discriminator to select a candidate while preserving user intent.A parallel branch supplies alternative rewrites from a frozen LLM; the stage aims to improve compositionality and multi-object fidelity.Source: §1 · p. 3
- SSPO iteratively evaluates RAPO-refined prompts using semantic, spatial, temporal, and task-specific feedback, then rewrites them to improve video quality without changing the generator.It stores feedback in a memory bank and uses average ranking to select a candidate for subsequent inference.Source: §1 · p. 3
- Stage 3 fine-tunes the rewriter LLM on original and SSPO-optimized prompt pairs, reusing iterative optimization patterns to improve generalization and reduce test-time computation.The paper reports that this fine-tuning accelerates convergence and extends RAPO++ across T2V architectures and downstream tasks.Source: §1 · p. 3
- 82.65% total score with LaVie and 80.75% with Latte on VBench; across five T2V models and five benchmarks, RAPO++ improves semantic alignment, composition, temporal stability, and physical plausibility.On T2V-CompBench, it also achieves state-of-the-art performance across challenging categories including consistent attribute binding and object interactions.Source: §1 · p. 3
- Analyses report that optimized prompts match training-prompt length and structure, fine-tuned LLMs improve multi-object fidelity, and ablations show complementary module contributions across LLM backbones.Physics-aware evaluators also yield gains in physical consistency and semantic alignment on PhyGenBench and VideoPhy over iterative refinement rounds.Source: §1 · p. 4
2 RELATED WORK
Prior prompt-optimization research largely targeted T2I or model-specific T2V rewriting, while diffusion-model test-time scaling has explored other ways to improve generation. The passage identifies iterative prompt refinement at test time as an underexplored direction.
2 RELATED WORK
- T2I and T2V generative models are sensitive to input prompts; well-performing prompts are often model-specific, coherent with training prompts, and misaligned with user input.Source: §2 · p. 4
- Prompt optimization is motivated by the mismatch between model-sensitive prompts and user inputs, while optimized prompts may still misalign with training prompts when LLM guidance is insufficient.The passage notes that T2I prompt optimization work,,, and T2V rewriting approaches, [16] do not necessarily align optimized prompts with training prompts.Source: §2 · p. 4
- Test-time scaling has been applied to diffusion generation through noise search, training and pruning strategies, and latent beam search, but not commonly through iterative prompt refinement.The passage identifies diffusion-model scaling work [50], [51], and states that few studies focus on scaling generative models by iteratively refining prompts.Source: §2 · p. 4
3 RAPO
RAPO combines relation-graph modifier retrieval, sentence refactoring, and prompt selection to enrich user prompts while aligning them with training-data formats.
3 RAPO
- RAPO uses word augmentation, sentence refactoring, and prompt selection to turn user prompts into training-data-aligned inputs for text-to-video generation.The selection module chooses between a refactored prompt and a naively augmented alternative.Source: §3 · p. 4
3.1 Word Augmentation Module
- The relation graph organizes training-prompt scenes as core nodes linked to subject, action, and atmosphere modifiers, which are retrieved to enrich user prompts.A sentence transformer and cosine similarity retrieve relevant scenes and modifiers.Source: §3.1 · p. 5
3.2 Sentence Refactoring Module
- The refactoring model is instruction-tuned on prompt pairs that preserve similar semantics while differing in format and length, mapping augmented prompts toward training prompts.The augmented-style prompts are generated by rewriting training prompts with an LLM.Source: §3.2 · p. 5, Table 2 · p. 6
3.3 Prompt Selection Module
- A discriminator selects between refactored and naively augmented prompts using labels derived from generated-video evaluations on prompt-specific dimensions.The training triples include the input prompt, both candidates, and the selection label.Source: §3.3 · pp. 5–6, Table 3 · p. 6
4 RAPO++
RAPO++ extends RAPO with iterative, feedback-guided test-time prompt refinement and LLM fine-tuning. SSPO evaluates generated videos across multiple dimensions, while optimized prompt pairs train the rewriter for later use.
4 RAPO++
- RAPO++ unifies RAPO, test-time SSPO, and LLM fine-tuning in a three-stage pipeline for test-time refinement and model-level enhancement.The framework builds on RAPO as Stage 1 and adds SSPO and fine-tuning as Stages 2 and 3.Source: §4 · p. 6
4.1 SSPO Mechanism
- SSPO assesses generated videos with semantic, spatial, and temporal feedback, optionally adding task-specific signals such as optical-flow-based motion and physical-plausibility checks.Feedback is passed to the rewriting module to guide subsequent prompt refinement.Source: §4.1 · pp. 6–7
- SSPO stores prior assessments and refinement outcomes in feedback memory, which informs reflection-driven rewriting across iterations.Candidate prompts are selected by their average rank across evaluation dimensions, rather than by a single metric.Source: §4.1 · p. 7
4.2 LLM Fine-Tuning
- Stage 3 instruction-tunes the rewriter on original and SSPO-optimized prompt pairs to reinforce generalization and mitigate local optima.The fine-tuned rewriter is deployed in the refinement pipeline.Source: §4.2 · p. 7
5 EXPERIMENTS
The experiments evaluate RAPO++ across varied text-to-video models, benchmarks, and task dimensions. Results report improvements in video quality, compositional generation, physical consistency, and semantic alignment, alongside ablations and a numeracy limitation.
5.1 Experimental Setup
- Evaluation spans open-source T2V models and complementary benchmarks assessing video quality, semantic alignment, composition, temporal consistency, and physical plausibility.The setup includes benchmarks for compositional reasoning and physical commonsense, among other dimensions.Source: §5.1 · pp. 8–9
5.2 Implementation Details
- Implementation uses Vimeo25M-derived data for RAPO and LLaVA-OneVision feedback with Qwen2.5-7B-Instruct for SSPO refinement.For physical-aware tasks, predicted optical flow supplies motion-field information as an additional assessment condition.Source: §5.2 · p. 9
5.3 Evaluation Results
- 82.65% with LaVie and 80.75% with Latte are RAPO++’s VBench total scores; RAPO and RAPO++ also outperform baselines on EvalCrafter and T2V-CompBench.RAPO++ leads across nearly all VBench submetrics, while RAPO and RAPO++ achieve the highest scores on the video-quality and compositional benchmarks.Source: §5.3 · p. 9, Table 7 · p. 10, Table 6 · p. 10
5.4 Extension to Physical-aware Video Generation
- Across PhyGenBench and VideoPhy, task-specific SSPO assessment improves physical consistency and semantic alignment over iterative refinement rounds.For HunyuanVideo on PhyGenBench, after four rounds physical consistency rises from 0.38 to 0.57 and semantic alignment from 0.24 to 0.42; on VideoPhy solid-solid interactions, its scores rise from 0.28 to 0.40 and from 0.41 to 0.65, respectively.Source: §5.4 · p. 11, Table 8 · p. 12, Table 9 · p. 13
5.5 Analyses
- RAPO prompts more closely match training-prompt length distributions, while fine-tuned LLM prompts improve multi-object and compositional generation.A qualitative example shows an optimized prompt more faithfully specifying a panda cashier scene than the initial prompt.Source: §5.5 · pp. 12–13
- LaVie’s four VideoScore dimensions rise across refinement iterations, while multiple inference passes plus VLM assessment increase runtime to roughly three times baseline.The reported average gains are 3.5% on VBench and 18.1% on T2V-CompBench across LaVie and Latte.Source: §5.5 · p. 13, §5.6 · p. 14
5.6 Ablation Study
- RAPO’s individual modules each improve performance, and their combination performs best; SSPO and LLM fine-tuning each help, with both together yielding the best results.The SSPO and fine-tuning ablation evaluates compositional metrics including consistent attribute binding, dynamic action binding, and object interaction.Source: §5.6 · pp. 13–14, Table 10 · p. 14, Table 12 · p. 14
5.7 Limitation
- RAPO++ still struggles with numeracy: videos may fail to generate the specified number of entities, such as five parrots or three giraffes.The paper attributes this limitation to T2V models blurring numerical information with broader semantics and SSPO lacking fine-grained count-aware feedback.Source: §5.5 · p. 12, §5.7 · p. 14
6 CONCLUSION AND FUTURE WORK
RAPO++ combines three prompt-optimization stages to improve text-to-video generation without changing the backbone, and the authors plan to extend its efficiency and task scope.
6 CONCLUSION AND FUTURE WORK
- RAPO++ refines prompts, iteratively improves them with feedback, and fine-tunes the LLM without changing the T2V backbone.The authors describe the framework as three-stage.Source: §6 · p. 14
- RAPO++ achieves superior compositionality, dynamics, and physical realism over existing methods.Source: §6 · p. 14
- Future work will target real-time inference efficiency and extensions to controllable video editing, multimodal scene synthesis, and text-to-3D generation.Source: §6 · p. 14