PaperlayerView exact revision on arXiv

Source-linked AI summary

ProactGen: Learning Proactive Clarification for Video Generation

Zhixiao Wang, Yang Chen, Yifan Xu, Chenbin Li, Yue Chang, Kai Long, Fengming Dong, Limin Wang

arXiv:2610.22608v1cs.CV

TL;DR

Underspecified prompts and costly video generation make matching user intent time-consuming, and rewriting cannot recover preferences omitted from the prompt. ProactGen learns to ask questions grounded in low-cost video drafts, using final-video alignment as its reward. It improves target-alignment gains over direct generation by at least 1.8× the strongest baselines.

  • Problem

    Costly video generations and underspecified prompts make it time-consuming to produce videos matching user intent, while rewriting cannot supply omitted preferences.

    Source: §1 · p. 1

  • Method

    ProactGen trains a questioning policy to use the initial prompt and a generated draft to ask questions, rewarding the policy with final-video alignment to a target.

    Source: §1 · p. 2

  • Results

    ProactGen’s target-alignment gain over direct generation is at least 1.8× that of the strongest baselines.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Effective questioning does not require final-quality drafts, so few-step drafts can keep clarification’s added cost low.

    Source: §1 · p. 2

    The evaluation uses simulated users rather than human participants, and the method’s capability is bounded by the underlying generator.

    Source: §3.1 · p. 4, ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Recent advances in video generation let users create high-quality videos from natural language descriptions, yet obtaining results that match user intent remains time-consuming: individual generations are computationally expensive, and underspecified prompts force repeated cycles of generation, inspection, and revision. Existing video prompt optimization methods improve generation through automatic rewriting, but cannot supply the ideas and preferences that were not in the initial description. We propose ProactGen, a video generation framework that learns proactive clarification, eliciting user intent through questions grounded in generated video drafts. Specifically, we train a vision-language model as a questioning policy with reinforcement learning: observing the initial prompt and a video draft, it learns to ask the questions most valuable to downstream generation, rewarded by the alignment between the final generated video and a target reference. We further find that effective questioning does not require drafts of final-output quality, and exploit this property by generating drafts with fewer denoising steps, keeping the added cost of clarification low. Experiments show that ProactGen improves target alignment over direct generation by at least 1.8x the gain of the strongest baselines. Moreover, a single round of ProactGen already exceeds the alignment that the strongest baseline reaches after four refinement rounds, in 29% of its time.

1 INTRODUCTION

Underspecified prompts and costly video generation make matching user intent time-consuming, while prompt rewriting cannot supply preferences absent from the prompt. ProactGen learns to ask draft-grounded questions, using low-cost drafts, and improves target alignment over existing approaches.

1 INTRODUCTION

  • At tens of denoising steps per full generation, costly video creation and underspecified prompts force repeated generation, inspection, and revision.Users often recognize missing or misinterpreted details only after seeing a result.Source: §1 · p. 1
  • Prompt rewriting improves expression of supplied information but cannot recover preferences absent from the initial description, while prior clarification selects questions from prompt text or a parsed belief state.The introduction identifies choosing questions from generated video drafts and learning from final-video outcomes as open in video generation.Source: §1 · pp. 1–2
  • ProactGen’s policy observes the prompt and a generated draft, asks scoped questions, and is trained with GRPO using final-video alignment to a target as reward.The user’s answers are merged into a refined prompt for final generation.Source: §1 · p. 2
  • Few-step drafts preserve the subject, layout, action, and camera cues needed for questioning, while later denoising mainly refines texture and detail.This lets ProactGen reserve the full denoising budget for the final video and keep clarification’s added cost low.Source: §1 · p. 2
  • ProactGen’s target-alignment gain over direct generation is at least 1.8× that of the strongest baselines; one ~16 s round exceeds four rewriting rounds or 15 clarification turns.Further rounds add further gains.Source: §1 · p. 2

2 RELATED WORK

Prior work improves prompts through rewriting or obtains user intent through interaction, but ProactGen distinguishes itself by learning questions grounded in a video draft and final-video alignment.

Prompt optimization

  • Video prompt optimizers rewrite prompts, sometimes using generated-output feedback, but remain limited to information already supplied by users (Gao et al., 2025a; Cheng et al., 2025; Ji et al., 2025; Yang et al., 2026a).For preferences absent from the prompt, these methods fall back on the model’s prior (Gao et al., 2025b; Long et al., 2026).Source: §2 · p. 3

Interactive clarification

  • Interactive clarification includes learned questions for downstream text answers (Andukuri et al., 2024; Dou & Liu, 2026; Zhang et al., 2025), image-generation user interaction, and video referring-segmentation clarification (Yang et al., 2026b).Proactive T2I uses heuristic or prompted questions selected from a belief graph, while other image methods learn candidate prompts or revise them from user input (Hahn et al., 2024; Nabati et al., 2024; Li et al., 2026; Wang et al., 2024).Source: §2 · p. 3
  • ProactGen grounds questions in the draft generated for the particular prompt and learns question selection from final-video alignment to the target.This contrasts with prior work selecting questions from request text or a parsed belief state, or learning from text responses or candidate choices.Source: §2 · p. 3

3 METHOD

ProactGen turns an incomplete prompt into a refined prompt by asking draft-grounded, scoped questions, then generates a full-budget video. It learns the questioning policy from final-video alignment to a hidden reference while keeping the other pipeline components frozen.

3.1 PROBLEM SETUP

  • The simulated-user setup treats a reference video as the user’s intent, while the policy and generator see only the incomplete prompt and generated draft.The reference is available to the simulated user and reward computation, not to the policy or generator.Source: §3.1 · pp. 3–4

3.2 DRAFT-GROUNDED CLARIFICATION PIPELINE

  • A reduced-budget draft grounds scoped, single-item questions; the reference-based user answers only asked, verifiable details, and a frozen refiner merges valid answers before full-budget final generation.Only the questioning policy is learnable; the generator, simulated user, and refinement model remain frozen.Source: §3.2 · p. 4
  • The same policy can be reused across rounds without separate multi-round training, while later rounds may observe either a draft or the previous full video.The method does not prescribe which later-round observation to use.Source: §3.2 · pp. 4–5

3.3 LEARNING TO ASK FROM GENERATION OUTCOMES

  • GRPO trains the policy using final-video alignment after each question set is answered and refined, with invalid questions penalized.Rewards are standardized within groups; the invalid-question penalty is capped per sample.Source: §3.3 · p. 5

3.4 LOW-COST DRAFTS FOR CLARIFICATION

  • Because subject, layout, action, and camera settle early, few-step drafts can expose prompt interpretation; a high-noise-heavy schedule preserves the full final-generation budget.The policy is trained on low-cost drafts from the start, and the final-generation budget is unchanged by the number of draft steps.Source: §3.4 · p. 5

4 EXPERIMENTS

The experiments evaluate whether learned, draft-grounded clarification improves video alignment and efficiency over generation, rewriting, and clarification baselines. ProactGen leads on the reported alignment measures, while ablations and examples examine the roles of training and draft observation.

4.1 EXPERIMENTAL SETUP

  • The evaluation uses 500 InstanceCap samples and 500 disjoint MiraData samples for MiraBench-500, with captions as prompts and video or caption-alignment metrics.The generator produces 49-frame videos; drafts use 8 denoising steps and final generation uses 20.Source: §4.1 · pp. 6–7
  • The comparison includes direct generation, prompt-rewriting systems, and Proactive T2I adapted from image generation to video.The two rewriting baselines have no access to the user; Proactive T2I uses the simulated user and reports 5- and 15-turn budgets.Source: §4.1 · p. 7

4.2 MAIN RESULTS

  • 0.020 DreamSim-DTW gain over Direct is more than twice Proactive T2I’s 15-turn gain of 0.009; ProactGen also leads all baselines on three InstanceCap metrics.On MiraBench-500, it also has the highest Overall, background, style, and camera alignment.Source: §4.2 · p. 7

4.3 ABLATION: REINFORCEMENT LEARNING AND DRAFT OBSERVATION

  • RL improves alignment by 0.010 with draft observation and 0.003 without; observing a draft helps only after training.The ablation compares trained and untrained policies, with and without an 8-step draft.Source: §4.3 · p. 7, Table 2 · p. 8

4.4 EFFICIENCY: ALIGNMENT PER UNIT OF TIME

  • At about 16 s, one ProactGen round raises alignment by 0.020, versus a 0.007 gain for one RAPO++-SSPO round costing about 21 s.Under the reported configuration, direct generation takes about 7 s; ProactGen’s draft, questioning, answering, and merging add about 9 s.Source: §4.4 · p. 8
  • 8-step drafts reach 0.678 after one round, close to 0.681 with 20-step drafts; they save about 11 s over four rounds at comparable alignment.The authors report that the advantage of low-cost drafts grows under heavier generation settings, including a 90 s full generation for 241 frames at 50 steps.Source: §4.4 · p. 8
  • In about 16 s, one ProactGen round exceeds RAPO++-SSPO after four rewrites (55 s) and Proactive T2I after 15 turns (242 s).The policy was trained only for one round; the authors report that its questioning skill transfers to later rounds.Source: §4.4 · p. 9

4.5 CASE STUDY

  • In two examples, questions target choices left open by the prompt but settled by the draft, including who holds the microphones and the camera angle.The authors also report that some answers are unknown or that the generator does not follow the refined prompt.Source: §4.5 · p. 9, Figure 4 · p. 9

5 CONCLUSION

ProactGen learns to elicit intent through draft-grounded questions, using low-cost drafts and final-video alignment as a reinforcement-learning signal. In simulated-user evaluation, it improves target alignment, while real-user evaluation and multi-round clarification within one denoising process remain future directions.

5 CONCLUSION

  • ProactGen’s target-alignment gain over direct generation is at least 1.8× that of the strongest video prompt optimization and proactive clarification baselines.One interaction costs about as much as one automatic-rewriting round yet exceeds baseline alignment after several rounds.Source: §5 · p. 9
  • The questioning policy observes the prompt and a generated draft, learns questions valuable to downstream generation through reinforcement learning, and uses low-cost drafts.The reward is alignment between the final video and target; extending the method to real users and integrating multiple clarification rounds into one denoising process remain future directions.Source: §5 · p. 9

AI USE STATEMENT

  • AI tools supported training-data preparation, simulated-user and prompt-merging components, code assistance, related-work organization, and language polishing.The authors state that they developed the research questions, method design, experimental protocol, and conclusions and take responsibility for the paper’s content.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • Evaluation uses public InstanceCap and MiraData data, and all experimental users are language-model simulations rather than human subjects.The authors state that ProactGen inherits the underlying generator’s potential for misuse and recommend its safety policies plus standard input and output filtering for deployment.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The authors plan to release the code, policy weights, model prompts, sample identifiers, noise seeds, and evaluation protocol configurations.The statement points to the paper’s methods, evaluation, data-construction, prompt, training, baseline, and timing sections for further details.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 TRAINING DATA CONSTRUCTION

The appendix describes how training prompts are constructed, how the 500-sample evaluation set is selected, and how the policy, simulated user, refinement model, and generator are configured.

A.1 TRAINING DATA CONSTRUCTION

  • Training prompts use 15,625 MiraData clips; captions are parsed into static and dynamic slots, and 2 to 5 slots are deterministically masked while retaining the subject.At least one static and one dynamic slot are masked; masking an action also masks dependent interaction and temporal-relation slots, and a language model rewrites the remaining facts without leaking masked information.Source: §A.1 · p. 13

A.2 MIRABENCH-500

  • The frozen 500-sample MiraBench evaluation set excludes training clips and overlapping clips, using official short captions as prompts and corresponding videos as references.The simulated user sees eight reference frames and the official five-field caption; metrics are five ViCLIP text–video alignments, with Overall based on dense-caption alignment.Source: §A.2 · p. 13

A.3 PROMPTS AND PROTOCOLS

  • The questioning policy receives the initial prompt and eight draft frames and may ask one to five English questions, each limited to one bounded visual fact.Questions are limited to 40 words and cannot answer themselves, bundle independent facts, list alternatives, or request a complete description.Source: §A.3 · p. 13
  • The simulated user answers using eight reference frames and the target caption, marking each response VALID, UNKNOWN, or INVALID; directly visible facts follow the video.The caption supplements temporal, motion, camera, and naming information sparse frames cannot confirm; the video takes precedence on conflicts, which are recorded.Source: §A.3 · p. 13
  • The refinement model combines the initial prompt with VALID question-answer pairs, preserving existing facts and adding no entity unsupported by either source.Unanswered questions remain missing information, and the initial prompt is used unchanged when no answer is VALID.Source: §A.3 · p. 13
  • Final videos use 20 denoising steps, while drafts use 8 steps; both use the same generator and other settings.Generation uses 49 frames at 24 fps and 832×480 resolution, with initial noise shared across conditions for each sample and across candidates in a GRPO group.Source: §A.3 · p. 13

A.4 TRAINING DETAILS

The policy is fine-tuned with GRPO, using aligned DreamSim similarity of final videos as its reward and a fixed final checkpoint.

A.4 TRAINING DETAILS

  • Training uses Qwen3-VL-8B with GRPO for one epoch of 1,954 steps, and the final checkpoint is fixed without early stopping or checkpoint selection.The reward is mean eight-frame aligned DreamSim minus an invalid-question penalty of 0.01 per question, capped at 0.05; the policy is fully fine-tuned with the vision encoder and projector frozen.Source: §A.4 · p. 13

A.5 EVALUATION METRIC

The appendix specifies DreamSim-DTW evaluation and its relationship to the training reward, alongside auxiliary metrics and the handling of failed samples.

A.5 EVALUATION METRIC

  • DreamSim-DTW scores video-reference similarity as one minus mean DreamSim distance along an optimal temporal alignment path over 49 uniformly sampled frames.The path has fixed endpoints and permits diagonal, horizontal, and vertical steps, without a warping window or slope penalty; unlike the referenced original metric, frames are not cropped to an animated region.Source: §A.5 · p. 14
  • The training reward averages DreamSim over eight aligned positions on pre-compression RGB, while evaluation samples and aggregates frames differently.Evaluation uses the same DreamSim weights and spatial preprocessing as the reward.Source: §A.5 · p. 14
  • Auxiliary measures include CogVideoX 3D-VAE latent distance and bidirectionally averaged DnS similarity; pipeline failures use the same sample’s Direct result and scores.This fallback keeps means and paired comparisons on the same fixed set of 500 samples; reported failures include 20 and 36 Proactive T2I cases at five and 15 turns, respectively, and one ProactGen parsing failure.Source: §A.5 · p. 14

A.6 PAIRED COMPARISONS

The paired comparisons report confidence intervals over the same 500 samples, while the baseline adaptations specify what kinds of clarification and rewriting are being tested. Some generator-specific methods or stages are excluded because they cannot be fairly reproduced on the authors’ generator.

A.6 PAIRED COMPARISONS

  • Table 3 reports paired differences with 95% confidence intervals from comparisons on the same 500 samples.Positive values favor the first method for DreamSim-DTW and DnS, but the second for 3D-VAE; parentheses indicate question turns or refinement rounds.Source: Table 3 · p. 14, §A.6 · p. 14

A.7 BASELINE ADAPTATION

  • Proactive T2I uses a belief graph to select questions, but its adapted baseline lacks generation grounding and learning.The adaptation uses eight reference frames; its simulated user can provide more information per turn than ProactGen, and some samples fail in parsing or dialogue.Source: §A.7 · pp. 14–15
  • RAPO++-SSPO runs four rounds of rewriting with semantic and physical-consistency feedback, selecting among the initial prompt and rewrites.It has no user access and tests automatic rewriting with generation feedback.Source: §A.7 · p. 15
  • RAPO and generator-specific RAPO++ stages are not compared because their retrieval or fine-tuning components cannot be fairly reproduced on this generator.The comparison uses only RAPO++’s generator-agnostic test-time stage, SSPO.Source: §A.7 · p. 15

A.8 EFFICIENCY MEASUREMENT

Efficiency is measured by per-sample stage times from the initial prompt through the final video. The single-round table breaks out method stages, with later-round times partly extrapolated.

A.8 EFFICIENCY MEASUREMENT

  • Reported end-to-end times sum stage means and exclude initialization, warmup, scoring, and archiving.Interactive-method timings include the simulated user’s answering stage.Source: §A.8 · p. 15
  • Some remote API and Proactive T2I language-model timings reuse earlier measurements, with the latter reconstructed from request dependencies as an estimate.Rounds three and four extrapolate the added time from rounds one to two rather than remeasuring every round.Source: §A.8 · p. 15
  • Table 4 reports per-sample stage times in seconds, averaged over 100 samples, for the single-round setting.“Questioning and merging” includes policy questions, simulated-user answers, prompt merging, and observation packing; “reference I/O” is processing the simulated user’s reference frames.Source: Table 4 · p. 15

A.9 ADDITIONAL RESULTS AND FAILURE CASES

Additional results characterize question answerability, training progress, and draft-step choices, alongside observed failure modes. The system prompts specify how the policy asks scoped questions and how the simulated user grounds its answers.

A.9 ADDITIONAL RESULTS AND FAILURE CASES

  • On 500 samples, ProactGen’s five questions per sample are 97.8% VALID, 1.9% UNKNOWN, and 0.3% INVALID.The trained text-only policy has 95.4% / 3.9% / 0.7%; the untrained draft-observing policy has 94.9% / 4.0% / 1.1%, and its VALID answers average 4.1 words versus 5.0 for ProactGen. Trained-policy questions are more often answerable from the reference and draw more informative answers.Source: §A.9 · p. 16
  • The 8-frame DreamSim development mean peaks at 0.679 at step 1400, while the final checkpoint reaches 0.674 at step 1954.The final checkpoint was fixed before training rather than selected by the development set.Source: §A.9 · p. 16
  • A 20-step-draft policy reaches 0.681 DreamSim after one round versus 0.678 for an 8-step-draft policy, a nonsignificant difference under the paired interval.These are independently trained policies.Source: §A.9 · p. 16
  • Observed failures include unanswerable questions, refinement errors that misattribute answers, and generators that omit correctly requested details.The reported cases include missing camera motion or a second person, indicating that part of the remaining gap lies in generator instruction following.Source: §A.9 · p. 16

B FULL SYSTEM PROMPTS

  • The questioning policy sees eight chronological frames of the frozen draft and asks one to five concise, bounded questions.Each question is limited to 40 words and may target visual or spatiotemporal details; the policy should ask rather than assume the draft is correct.Source: B FULL SYSTEM PROMPTS · p. 16
  • The simulated user answers from eight time-aligned reference frames and written intent, using the video as primary evidence for observable facts.Answers to valid questions are limited to 10 words; UNKNOWN is used when the intent and video do not determine an answer.Source: B FULL SYSTEM PROMPTS · p. 17
  • The refinement prompt combines the initial prompt with valid clarification pairs while preserving existing details unless an answer explicitly contradicts them.It must not add entities mentioned only in a question when the answer negates them.Source: B FULL SYSTEM PROMPTS · p. 17
Loading 2610.22608v1…