No Corners Cut: State-Grounded Transitions for Mid-Stream Prompt Switches in Video Generation

• Existing systems can preserve visual smoothness yet still duplicate objects, break physical contact, or jump abruptly between states after a prompt switch.The paper calls this “corner cutting”: moving toward the new instruction without accounting for a semantically plausible transition path.Source: §1 · pp. 1–2
• SEGUE uses a training-free planner to turn the realized visual boundary into role-organized segue prompts, then uses SPANDMD to train causal generators to execute them.SPANDMD evaluates each active prompt with the complete rollout as context but retains its DMD contribution only on that prompt’s assigned span.Source: §1 · p. 2
• SEGUE achieves the strongest overall performance across segment- and transition-level OpenTrans-360 evaluation, ranks first on all eight transition metrics, and ranks first on four of six StreamAV-Bench instruction-response metrics.The benchmark evaluates 360 prompt streams containing 1,800 mid-stream switches, and its checklist metrics achieve an average Spearman ρ of 0.87 against human judgments.Source: §1 · pp. 2–3
• OPENTRANS-360 evaluates both within-segment generation quality and switch-level transition quality, with checklist metrics achieving an average Spearman ρ of 0.87 against human judgments.Source: §1 · p. 3

• Autoregressive video methods improve causal generation and streaming, while interactive systems add prompt updates, cache management, or memory for continuity under evolving prompts.The paper focuses on updates that continue the current scene rather than introduce a cut or reset.Source: §2 · p. 3
• Temporal planners and multi-event methods generally schedule prompts or boundary conditions before generation, whereas SEGUE plans online switches from the realized visual state.The same user prompts can require different transitions depending on what has actually been generated when the switch occurs.Source: §2 · p. 3
• OPENTRANS-360 evaluates whether a transition resolves the preceding event and reaches the requested event through a plausible intermediate process.Other cited benchmarks cover perceptual quality, motion, alignment, compositional semantics, state consistency, temporal consistency, or responsiveness.Source: §2 · p. 3
• SEGUE’s overview pairs a state-grounded planner that compiles a dependency-aware segue schedule with a causal generator trained using span-scoped DMD residuals and full-rollout context.Source: Figure 2 · p. 4

3.1 PROBLEM FORMULATION

• A direct switch applies the new prompt to a continuation whose realized state may not meet the target event’s prerequisites, leaving intermediate changes unspecified and inviting corner cutting.The prior generated history has already been shown and cannot be revised.Source: §3.1 · p. 4

3.2 STATE-GROUNDED SEGUE PLANNING

• The planner grounds its decisions in the latest frame, extracting event progress, entities, contacts, pose, and spatial relations, and follows observed state when it conflicts with the old prompt.TD-Schema selects required TERMINATE, RELEASE, ALIGN, or ENTRY roles; the compiler orders dependent operations, merges compatible changes, and rejects invalid or over-budget schedules.Source: §3.2 · pp. 4–5
• Each segue prompt receives a block span, after which control returns to the unchanged user prompt; a newer prompt arriving mid-schedule discards the unexecuted suffix and triggers replanning.The planner changes text conditioning and can run with a frozen generator.Source: §3.2 · p. 5

3.3 SPANDMD: SPAN-SCOPED DISTILLATION WITH FULL-ROLLOUT CONTEXT

• SPANDMD trains short-lived prompt schedules because long-lived-prompt generators may skip or blend stages, while target-only rollout scoring can reward shortcuts and clip cropping removes temporal context.The old prompt, each segue prompt, and the target prompt are supervised on their respective spans while both score models process the complete noised rollout.Source: §3.3 · pp. 5–6
• Training schedules come from a text-only LLM planner, whereas inference schedules are produced online by the state-grounded planner from the realized visual state.Source: §3.3 · p. 6

Benchmark design

• The benchmark contains 360 six-prompt streams, each producing five switches in a 60-second video, for 1,800 switches total.Fixed 10-second prompt intervals standardize when prompts arrive, not how long transitions take; each switch tests continuation from the model’s own realized state.Source: §4 · p. 6

Benchmark construction

• Streams cover human-centric, object-centric, and compositional transitions, are filtered by automatic checks and human inspection, and contain no scene cuts.Benchmark prompts are disjoint from the VidProM prompt pairs used to construct training schedules.Source: §4 · p. 6

Metrics

• Eleven metrics assess segment and transition quality; the Overall score is their unweighted mean, and higher scores are better.Transition metrics include old-state exit, new-state entry, semantic update fidelity, and transition pacing, evaluated in metric-specific windows around each switch.Source: §4 · p. 6, Table 1 · p. 7

Human agreement

• Across eleven metrics, benchmark scores correlate with human win rates at a mean Spearman ρ of 0.87.The blinded pairwise study covered 30 stratified cases.Source: §4 · p. 6

5.1 SETTINGS

• Experiments use a LongLive-initialized Wan2.1-T2V-1.3B generator and compare two closed-source and ten open-source methods on shared prompt streams.The planner is used without fine-tuning; training schedules contain approximately 45k precomputed segue prompts.Source: §5.1 · p. 7

5.2 COMPARISON RESULTS

• SEGUE scores 0.887 overall versus 0.866 for HappyOyster and ranks first on all eight OPENTRANS-360 transition metrics.The paired difference is 0.021 (95% case-bootstrap CI [0.017, 0.031]); the table reports competitive segment quality as well.Source: §5.2 · p. 7
• On the bell switch, SEGUE lifts the existing bell while TempAct and Anchor Forcing duplicate it; blinded preferences exceed 50% against all twelve baselines.LongLive, IAMFlow, and Self-Forcing do not lift the bell in this example.Source: §5.2 · pp. 7–8, Figure 3 · p. 8
• On StreamAV-Bench, SEGUE ranks first on four of six instruction and response metrics and improves PVC by 7.1% over the best baseline.Visual quality is on par with the best baselines: 0.603 VA and 2.838 VQ versus 0.605 and 2.840.Source: §5.2 · p. 8

5.3 ABLATION STUDY

• With planning and vanilla DMD, MBS rises from 0.824 to 0.869, OSE from 0.717 to 0.759, and SUF from 0.747 to 0.790.Planner additions also improve all six evaluated transition metrics for both frozen Self-Forcing and LongLive generators.Source: §5.3 · pp. 8–9, Table 2 · p. 9
• Removing SPANDMD lowers all six reported transition metrics, including OSE from 0.802 to 0.759, MBS from 0.897 to 0.869, and NSE from 0.890 to 0.865.This ablation jointly changes score-network conditioning and temporal residual assignment, so it does not isolate span masks; its visual failures include interpenetration, identity changes, and a disappearing terrier.Source: §5.3 · pp. 8–9, Figure 4 · p. 8, Table 2 · p. 9

Conclusion

• SEGUE uses segue prompts organized by TD-Schema, while SPANDMD trains with span-specific supervision and full-rollout context.The paper reports its highest overall score and first place on all eight transition metrics on OpenTrans-360, plus improved instruction following and responsiveness with competitive visual quality on StreamAV-Bench.Source: §6 · pp. 9–10

AI-use disclosure

• Generative AI tools assisted dataset construction and filtering, and were components of the pipeline, including the VLM planner used at inference.The paper says these roles are described in the corresponding method and experimental sections.Source: AI USE STATEMENT · p. 10
• Authors report that they reviewed AI-assisted data checks, code, and manuscript edits, and take responsibility for the final work and artifacts..Source: AI USE STATEMENT · p. 10

Appendix contents

• The appendix lists a user study in A.1.Source: A APPENDIX · p. 13
• The appendix lists limitations and future work in A.2.Source: A APPENDIX · p. 13
• The appendix lists additional StreamAV-Bench results in A.3.Source: A APPENDIX · p. 13

A.1 USER STUDY

• Twelve participants assessed 30 stratified cases, with one switch selected per case before model outputs were inspected.Each comparison used the same switch and a clip window from 2 seconds before to 8 seconds after the update.Source: §A.1 · p. 14
• Participants viewed synchronized, anonymized clips and separately rated visual quality, transition naturalness, and overall preference.Method identities, benchmark scores, and planner-generated waypoints were hidden; overall preference was judged directly rather than averaged from the other criteria.Source: §A.1 · p. 14
• 1,039 of 1,080 planned criterion-specific comparisons yielded a resolved majority judgment, and repeat consistency was 89%.The remaining 41 comparisons were excluded; pairwise agreement among original annotators was 83%.Source: §A.1 · p. 14

Physical prerequisites in planning

• Role ordering does not ensure physical prerequisites: the planner may release the lantern before explicitly establishing cart support.The generated placement can still appear complete, potentially because the generator compensates for the underspecified plan.Source: §A.2 · p. 14, Figure 7 · p. 16

Partial observation and execution reliability

• The planner sees only the latest decoded frame, which may omit motion history or occluded interactions, and valid plans may not be faithfully executed.Physical consistency and identity preservation remain limited by the underlying generator.Source: §A.2 · pp. 14–15

Training cost

• SPANDMD distillation cost increases with the number of supervised stages because each active condition requires a separate score-network evaluation.The authors suggest more efficient multi-condition score estimation and adaptive supervision schedules as possible ways to reduce this overhead.Source: §A.2 · p. 15

Training–inference planning gap

• Training schedules come from a text-only LLM while inference uses a VLM grounded in the realized state, so some state-dependent transitions may be underrepresented during training.The authors report generalization to visually grounded inference schedules and propose generating training segue prompts from real video data.Source: §A.2 · p. 16

A.3 RESULTS ON STREAMAV-BENCH

• 7.1%: PVC improves over the previous best on StreamAV-Bench, where SEGUE ranks first on four of six semantic-related metrics.The best scores are VIF 3.667, VID 0.301, PVC 4.581, and PVRL 10.311.Source: §A.3 · p. 16
• VA 0.603 and VQ 2.838 closely match the best results of 0.605 and 2.840, respectively.SEGUE also has the lowest VA-D (0.021) and second-lowest VQ-D (0.247).Source: §A.3 · p. 16

A.4 EFFECT OF VISUAL GROUNDING

• The comparison held the generator, TD-Schema, transition budget, prompt update, generated history, and generation noise fixed.Five switches with two seeds were evaluated on each of 25 selected cases; 244 of 250 pairs were retained.Source: §A.4 · pp. 16–17
• 4.9% TP, 4.2% OSE, and 3.7% NSE are the largest relative gains when the planner receives the current frame.Visual grounding improves all six evaluated metrics across 244 matched pairs from 25 OpenTrans-360 cases.Source: §A.4 · p. 17, Table 5 · p. 17
• In the pastry-chef example, the grounded plan releases the flask before reaching for flour, while the text-only plan omits that release.In the watchmaker example, the grounded plan lowers the hands and bends toward the terrier before attempting the lift.Source: §A.4 · p. 17

A.5.1 INPUTS AND PLANNING OUTPUT

• The planner grounds its transition plan in the latest frame, preceding prompt, updated prompt, and transition budget; qualitative examples show grounding adds needed preparatory actions.The text-only plans omit releasing the flask before reaching for flour and bending toward the terrier before attempting a lift.Source: §A.5.1 · p. 17, Figure 8 · p. 18

A.5.2 STATE-DEPENDENT CONSTRAINTS IN TD-SCHEMA

• TD-Schema assigns preparatory roles to ending conflicting activity, removing obstructing interactions, establishing prerequisites, and introducing absent entities; EXECUTE hands control to the target prompt.RELEASE applies only to dependencies that obstruct the target event, while EXECUTE is not an additional segue prompt.Source: §A.5.2 · pp. 18–19, Table 7 · p. 18
• Preparatory roles are conditional, not a fixed chain: instantiated roles follow precedence, ALIGN may repeat, and ALIGN and ENTRY can occur in either scene-appropriate order.Already-satisfied conditions require no corresponding TERMINATE, RELEASE, or ENTRY role.Source: §A.5.2 · p. 19
• Segue prompts describe observable changes from the grounded source state, preserve unrelated scene elements, and divide a budget of up to four generation blocks before the target prompt resumes.The plan specifies each prompt's role, action, duration, start state, and end state.Source: §A.5.2 · p. 19, §A.5.3 · p. 20

A.5.3 PLANNER PROMPT SPECIFICATION

• The prompt specification directs the planner to omit satisfied roles, preserve continuity, describe changes rather than completed states, and leave the target event to the original updated prompt.Its templates are presented as a method-level specification, with implementation-specific serialization and bookkeeping omitted.Source: §A.5.3 · pp. 19–20

A.5.4 DETERMINISTIC COMPILATION AND FAILURE HANDLING

• The compiler orders roles, merges compatible actions, validates the resulting plan, and rejects plans that remain inadmissible.ALIGN and ENTRY share a precedence level, so their relative order follows the planner's proposal when admissible; distinct prerequisite actions can remain separate.Source: §A.5.4 · pp. 20–21
• Over-budget plans undergo duration normalization, which assigns rounding remainder to the longest rescaled prompt; plans that cannot meet the block budget with positive integer durations are rejected.Accepted segue prompts are repeated for their assigned durations, after which control returns to the unchanged target prompt.Source: §A.5.4 · p. 21
• Rejected or empty plans, malformed outputs, and planner failures use the same fallback: the updated prompt conditions the reserved transition blocks without an added intermediate action.The planner is not automatically queried again after rejection.Source: §A.5.4 · p. 21

A.6 ANALYSIS OF SPANDMD

• Across 200 videos, SPANDMD reduced color error throughout s1–s5 and beat vanilla DMD at most sampled times, especially s1–s4; random updates stayed near zero.The comparison used equal-magnitude latent updates from shared initial videos with model parameters frozen. SPANDMD keeps each prompt’s residual within its assigned span while using the full rollout as context; the conditional-score interpretation is approximate, and the ablation does not isolate the benefit of hard masking.Source: §A.6 · pp. 21–22, Figure 9 · p. 23

A.7 BENCHMARK DESIGN, CONSTRUCTION, AND EVALUATION

• OpenTrans-360 evaluates whether prompt updates resolve prior activity, establish prerequisites, and begin new events coherently, beyond prompt fulfillment and visual smoothness alone.Its semantic-transition dimensions complement measures of content quality and boundary continuity.Source: §A.7 · p. 23

A.7.1 DATA DISTRIBUTION

• OpenTrans-360 contains 360 six-prompt cases and 1,800 switches across eight transition types, targeting updates within persistent settings and excluding scene cuts.The switch types are derived from expected step annotations and scenario tags, not observed generated transitions.Source: Figure 10 · p. 24, §A.7.1 · p. 24

A.7.2 LLM-ASSISTED DATA CONSTRUCTION

• LLM-assisted construction expands structured case specifications into six prompts with shared scene context, then validates caption structure, compatibility, and category-specific constraints.Invalid candidates can be regenerated up to three attempts; expected transition annotations come from event templates, while the visual evaluator scores generated frames and prompts without requiring a prescribed waypoint chain.Source: §A.7.2 · pp. 24–25

A.7.3 METRIC DEFINITIONS AND COMPUTATION

• The benchmark defines three segment metrics and eight switch metrics, covering prompt fidelity, motion and physical plausibility, prior-event resolution, preparation, update fidelity, and pacing.The five semantic-transition dimensions are conceptual categories covered by these metrics, not additional scores.Source: §A.7.3 · p. 25

A.8 LLM-ASSISTED CONSTRUCTION OF TRANSITION TRAINING DATA

• For SpanDMD training, an offline collection of approximately 45k segue schedules is generated from source–target prompt pairs, with up to four intermediate prompts in a four-block budget.Automatic checks screen plans for structure, role consistency, duration, and caption form; at inference, the visual planner creates schedules from the realized scene state.Source: §A.8 · p. 26

A.9 HUMAN ALIGNMENT OF THE EVALUATION METRICS

• In the blinded metric-alignment study, 831 of 858 comparisons yielded resolved majorities; repeat consistency was 93% and original-annotator agreement was 88%.The study also compared method-level human win rates with benchmark scores using Spearman correlations; the supplied passages do not report their values.Source: §A.9 · p. 28, Table 14 · p. 28

A.10 ADDITIONAL IMPLEMENTATION DETAILS

• The generator starts from the LongLive base checkpoint without long-video tuning; LoRA adapters train on student and fake-score transformer linear layers while their backbones, text encoder, and video autoencoder remain frozen.Each block contains three latent frames, or 12 video frames; the four-block segue budget is approximately three seconds, and planning runs asynchronously alongside generation.Source: §A.10 · p. 28
• MBS evaluates whether trajectories, poses, and contact relationships remain coherent across the boundary, using four samples around the switch without text instructions.Its ten binary criteria include coherent contact, no ghosting, a single trajectory, and natural motion or pose change between frames 2 and 3.Source: §A.10 · pp. 31–32, Table 20 · p. 31
• OSE and NSE assess whether the old process ends understandably and the new event begins through visible preparation from a reachable preceding state.Both use adjacent prompts and a frame-1 reference; OSE samples four frames over (t, t + 1.5), while NSE samples four over (t, t + 5). Each has ten binary criteria.Source: §A.10 · pp. 32–33, Table 21 · p. 32, Table 22 · p. 32
• SP checks persistent-subject identity, while ScP checks instruction-consistent environmental change and coherence of spatial context and unrelated scene elements.Both use two pre-switch and two post-switch samples with adjacent prompts; SP is evaluated only on 300 subject-bearing cases, excluding 60 subject-free cases, and ignores newly introduced subjects.Source: §A.10 · pp. 33–34, Table 23 · p. 33, Table 24 · p. 33
• SUF assesses whether visible changes match the prompts’ semantic difference while preserving shared meaning, and TP assesses response onset, transition duration, and progress rhythm.SUF samples two pre-switch and four post-switch frames; TP samples two pre-switch and six post-switch frames. Both use adjacent prompts and have ten binary criteria.Source: §A.10 · p. 34, Table 25 · p. 34, Table 26 · p. 34