PaperlayerView exact revision on arXiv

Source-linked AI summary

FrameMorrow: Future-guided Frame Selection with Prospective Tokens for Long-Horizon Video Generation

Bo Yin, Xiaobin Hu, Jiaqi Zhao, Shuicheng Yan

arXiv:2609.38839v1cs.CV

TL;DR

Long-horizon video generation needs to preserve coherent visual details, but keeping all history is costly and redundant, and current relevance may not match future needs. FRAMEMORROW uses prospective tokens to select explicit historical frames, and evaluations across five benchmarks and 11 models report improvements in long-range consistency, visual quality, and action alignment.

  • Problem

    Long-horizon video generation must preserve visual coherence over time, but retaining all history is costly and redundant, while current relevance may differ from what future generation needs.

    Source: §1 · p. 1

  • Method

    FRAMEMORROW predicts prospective tokens as compact representations of future information needs and uses them to select explicit historical frames for compatible generators.

    Source: §1 · p. 2

  • Results

    Experiments across five benchmarks and 11 models report improvements in long-range consistency, visual quality, and action alignment.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The results support selecting historical information according to future needs rather than current relevance alone.

    Source: §5 · p. 10

    The future-grounded relevance proxy does not measure incremental utility beyond recent context or guarantee coverage across the continuation.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Long-horizon video generation requires models to effectively leverage an increasingly long generation history. As the generated history grows, retaining all previous content becomes increasingly expensive and redundant, making effective historical selection essential. Existing approaches often determine historical relevance based on the current content. However, information relevant to the present is not necessarily useful for future generation, while seemingly less relevant history may become important later. Our key insight is that historical information should be selected according to its relevance to future information needs. Capturing these needs does not require generating the full future; instead, a compact representation of what becomes important next is sufficient to guide historical selection. Building on this insight, we propose FrameMorrow, a prospective frame selector that predicts a small set of prospective tokens representing future information needs and uses them to identify relevant information from history. FrameMorrow selects explicit historical frames rather than model-specific internal states, enabling plug-and-play integration across diverse generators, including closed-source models, with little additional inference cost. We evaluate FrameMorrow across five benchmarks and 11 generative models spanning long-video generation, interactive generation, and action-conditioned world models. Extensive experiments demonstrate consistent improvements in long-range consistency, visual quality, and action alignment across diverse generation settings.

1 INTRODUCTION

Long-horizon video generation must preserve visual details across time, but retaining all history is costly and redundant. FRAMEMORROW selects history for its relevance to future information needs, using prospective tokens and explicit frames; experiments span five benchmarks and 11 generative models.

1 INTRODUCTION

  • Because generation history grows while earlier details can leave the model’s input window, preserving relevant history is central to long-horizon coherence.Keeping all history increases computation and introduces redundant context.Source: §1 · p. 1
  • Current-context matching can favor history similar to the present even when less-currently relevant information may better support future generation.The motivation figure contrasts current-view matching with selecting earlier history relevant to the upcoming generation, which helps preserve previously observed visual details.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • FRAMEMORROW predicts a small set of prospective tokens that represent future information needs and guide selection of relevant historical frames.The tokens are a compact proxy for future needs, so selection does not require generating the full future.Source: §1 · p. 2
  • The paper formulates future information needs as a condition for frame selection, addressing the limits of selecting history only by current content.Current-relevant frames may be unhelpful for future generation.Source: §1 · p. 2
  • Selecting explicit historical frames rather than model-specific internal states makes FRAMEMORROW plug-and-play across generators, including closed-source models, with little additional inference cost.The design selects only a few prospective tokens and historical frames.Source: §1 · p. 2
  • Five benchmarks and 11 generative models show improvements in long-range consistency, visual quality, and action alignment.The evaluations cover long-video generation, interactive generation, and action-conditioned world models.Source: §1 · p. 2

2 RELATED WORK

Related work spans long-horizon generation, historical-information retrieval and memory, and learned frame selection. FRAMEMORROW differs by selecting explicit historical frames for future information needs rather than building a generator backbone or relying on observed content or model-specific memory.

Long-Horizon Video Generation

  • Long-horizon video work develops generation backbones and memory capabilities, whereas FRAMEMORROW provides a plug-and-play selector for existing generators.The cited examples include streaming synthesis, reducing the training–inference gap, real-time generation, and action-controlled interactive worlds.Source: §2 · p. 3

Historical Information Selection

  • Historical-information methods retrieve visual context or manage internal caches, while FRAMEMORROW selects history according to future information needs.Prior approaches may use recent content, prompts, planned actions, or model-specific memory states to determine relevance.Source: §2 · p. 3

Learning to Select Frames

  • Learned frame selectors target evidence for questions or reasoning over fully observed videos, while FRAMEMORROW predicts prospective tokens for not-yet-generated content.Those tokens represent future information needs and guide selection of relevant historical frames.Source: §2 · p. 3

3 METHOD

FRAMEMORROW predicts compact representations of future information needs, uses them to select explicit historical frames, and trains their rankings against future content.

3.1 PROSPECTIVE FRAME SELECTION

  • The selector distinguishes recent context from eligible long-term history and uses the available rollout condition, which contains no later input, feedback, or observation.Recent context informs retrieval without consuming the long-term memory budget; selected frames are timestamp-ordered before reaching the backbone.Source: §3.1 · p. 3

3.2 PROSPECTIVE TOKENS

  • A causal Transformer autoregressively predicts prospective tokens from history, recent context, and the rollout condition; each token can condition on preceding tokens.The tokens query eligible historical frames; smooth-maximum aggregation lets a frame score highly by matching any token, without explicitly enforcing diversity.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3 FUTURE-GROUNDED RANKING DISTILLATION

  • A frozen DINOv2 teacher ranks historical frames by correspondence with realized future content, and listwise and pairwise losses train the selector.The correspondence proxy is not a measure of incremental utility beyond recent context or a guarantee of coverage; continuation-based target construction is removed at inference.Source: §3.3 · pp. 4–5

3.4 PLUG-AND-PLAY INTEGRATION

  • Selected explicit frames pass through each frozen generator’s native conditioning mechanism, requiring no newly trained conditioning module.A fixed K bounds the additional frames supplied, while selector processing depends on candidate count, recent-context length, and condition length.Source: §3.4 · p. 5

4 EXPERIMENTS

FRAMEMORROW is evaluated across five benchmarks and multiple generator families, with results spanning long-video, interactive-video, closed-source, and action-conditioned generation. Ablations examine future supervision, frame and token budgets, selection strategies, and inference overhead.

4.1 EXPERIMENTAL SETUP

  • The evaluation covers five benchmarks and long-video, interactive-video, closed-source, and action-conditioned generation, using metrics tailored to each setting.Experiments use 10K OpenVidHD videos for text-conditioned generation and 10K chunk-aligned Sekai Game-Walking examples for action-conditioned world models.Source: §4 · p. 6, §4.1 · p. 6

4.2 MAIN RESULT

  • FRAMEMORROW achieves the best average rank on all three long-video backbones: 1.33 for Self-Forcing, 1.25 for LongLive 1.0, and 1.33 for Causal Forcing.With Self-Forcing, imaging quality rises from 62.22 to 68.47 and dynamic degree from 51.72 to 64.56; LongLive-RAG retains higher background and motion scores on Self-Forcing.Source: §4.2 · pp. 6–7
  • For interactive video, FRAMEMORROW improves quality and consistency in single- and multi-shot settings; Self-Forcing consistency rises from 84.95 to 89.30, while ShotStream gains 4.01 consistency points.Self-Forcing’s segment-wise CLIP-score gain grows from 0.16 in the first 10 seconds to 2.99 in the last 10 seconds; ShotStream’s aesthetic score decreases by 0.56.Source: §4.2 · p. 7
  • With the same reference budget, FRAMEMORROW improves consistency by 1.29 points on Seedance 2.0 and 1.33 points on Kling O3, and outperforms uniform selection on all three reported metrics for both.The evaluation uses the models’ public reference-conditioning interfaces.Source: §4.2 · pp. 7–8, Table 3 · p. 7
  • Across Matrix-Game 3.0, WorldMem, and YuMe 1.5, FRAMEMORROW improves visual measures and action alignment while largely preserving motion quality.Action alignment increases by 0.019 for Matrix-Game 3.0, 0.023 for WorldMem, and 0.029 for YuMe 1.5.Source: §4.2 · p. 8, Table 4 · p. 8, Figure 5 · p. 8

4.3 ANALYSIS AND ABLATIONS

  • Future distillation adds 3.10 points over recent-context supervision, while four autoregressive prospective tokens improve consistency to 89.30 and exceed parallel prediction by 1.00 point.Future distillation reduces the oracle gap from 4.40 to 1.30 points; increasing the autoregressive token count from four to eight adds only 0.30 points, with no further gain at sixteen.Source: Figure 6 · p. 9, §4.3 · p. 9
  • FRAMEMORROW reaches 88.00 with two selected frames, exceeding uniform sampling with sixteen frames at 87.10; increasing K from 4 to 8 adds only 0.40 points.There is no further improvement at K = 16.Source: §4.3 · p. 9
  • FRAMEMORROW outperforms five alternative frame-selection strategies, and its inference overhead is measured across three models under matched hardware and sampling settings.The comparison strategies include recent-frame selection, uniform sampling, context matching, prompt matching, and direct scoring; overhead is measured with K = 4, M = 4, and selection refreshed every 5 seconds.Source: §4.3 · p. 9, Table 5 · p. 9, Table 6 · p. 9

5 CONCLUSION

The conclusion presents FRAMEMORROW as a prospective-token selector for relevant historical frames, with explicit-frame integration across generators. Experiments across five benchmarks and 11 models show improvements in consistency, visual quality, and action alignment.

5 CONCLUSION

  • FRAMEMORROW uses prospective tokens to predict future information needs and select relevant historical frames, with experiments supporting selection by future needs rather than current relevance alone.Its explicit-frame interface supports lightweight, plug-and-play use across diverse generators.Source: §5 · p. 10

A.1 SELECTOR ARCHITECTURE

The appendix describes the selector architecture, training targets and losses, optimization, text-conditioned training tuples, and segment annotation. The design retains each backbone’s native recent window while selecting additional eligible historical frames.

A.1 SELECTOR ARCHITECTURE

  • Frozen visual and condition encoders feed learned projections, a causal Transformer, and ranking query/key projections; compatible text-conditioned backbones share one checkpoint.Action-conditioned experiments use a separate checkpoint with the same architecture.Source: §A.1 · p. 13, Table 10 · p. 15

A.2 RANKING TARGETS AND LOSSES

  • The teacher uses cosine similarities between normalized historical-frame and realized-future features to create ranking targets; future observations provide detached targets only.The teacher samples eight continuation frames at two fps over the next four seconds; pairwise softplus loss applies to qualifying teacher-score gaps, while the listwise term remains active.Source: §A.2 · p. 13

A.3 OPTIMIZATION AND COMPUTE

  • Text- and action-conditioned selectors are trained independently on precomputed frozen-encoder features, with validation listwise loss selecting the final checkpoint.The stated planning budget is 12 GPU-hours per selector on two A100 80GB GPUs, excluding annotation and feature extraction.Source: §A.3 · p. 13

B.1 TEXT-CONDITIONED TRAINING TUPLES

  • Text-conditioned training uses 10K OpenVidHD videos split into two-second annotation units, while tuples include recent context, eligible history, a next-rollout condition, and teacher-only continuation.Recent context comprises eight frames from the preceding four seconds; up to 128 older frames are sampled before that window.Source: §B.1 · p. 14

B.2 SEGMENT ANNOTATION AND PROMPT CONSTRUCTION

  • Segment annotations preserve visible identities, scene attributes, actions, and state changes; malformed outputs are regenerated once and then excluded if still invalid.The annotation prompt asks for subject, scene, action, state, change, and a concise segment prompt, while forbidding inference of unseen events or future actions.Source: §B.2 · pp. 14–15

B.3 ACTION-CONDITIONED TRAINING TUPLES

The action-conditioned training data pairs observed history with pseudo-actions and realized continuations, while the shared selector supplies timestamped historical frames through each backbone’s native conditioning interface. Matched controls keep the history, budget, and conditioning adapter fixed to isolate frame-selection quality.

B.3 ACTION-CONDITIONED TRAINING TUPLES

  • Action-conditioned examples pair observed history and context with next-chunk pseudo-actions and realized continuations; camera-motion labels use relative translation and rotation with specified thresholds.Translation components activate above 0.2 after speed normalization; yaw and pitch use a 2°/s threshold, and simultaneous labels are retained.Source: §B.3 · p. 15

C.1 SHARED SELECTION AND BACKBONE-SPECIFIC CONDITIONING

  • One selector checkpoint per condition modality returns four explicit frames, which backbone adapters convert into each generator’s native reference representation.Generator activations do not feed back into the selector, and integration requires no generator or conditioning-module training.Source: §C.1 · p. 15

C.2 TEMPORAL POSITIONS AND CACHE MANAGEMENT

  • Adapters preserve selected-frame timestamps or map frames to temporal reference slots; refreshes replace historical slots while retaining native recent context.The four-frame budget fixes additional observations, not latent-token or KV-cache size across backbones.Source: §C.2 · p. 15

C.3 MATCHED INTEGRATION CONTROLS

  • Matched controls share eligible history, budget, and conditioning adapter, contrasting current-context matching and prompt matching with FRAMEMORROW to isolate selection quality.Comparisons with the unaugmented backbone also include the benefit of adding historical references.Source: §C.3 · p. 16

D EVALUATION PROTOCOLS

The evaluation uses matched generation settings and task-specific protocols across long-video, interactive, closed-source, and action-conditioned settings. Additional ablations examine the visual teacher, ranking objective, continuation horizon, and query-score aggregation.

D.1 LONG VIDEO GENERATION

  • Long-video evaluation generates 60-second videos from 128 prompts, scores six VBench-Long dimensions, and reports average rank within each backbone.Settings are matched within each backbone, evaluation covers the full video, and frame selection refreshes every five seconds using information available at that time.Source: §D.1 · p. 16, D EVALUATION PROTOCOLS · p. 16

D.2 INTERACTIVE VIDEO GENERATION

  • Interactive evaluation covers 100 single-shot and 100 multi-shot cases with six ten-second condition intervals, measuring segment-wise prompt alignment and recurring-subject consistency across shots.Multi-shot quality also measures temporal quality within shots, avoiding penalties for intended scene changes.Source: §D.2 · p. 16

D.3 CLOSED-SOURCE GENERATION

  • Table 11 is identified as the action-alignment rubric for the evaluation.Source: Table 11 · p. 17

D.4 ACTION-CONDITIONED WORLD MODELS

  • Action-conditioned world-model evaluation tests three models on 60 trajectories each, measuring visual properties and whether rollouts follow supplied controls.The judge scores observable action compliance rather than physical correctness, and malformed outputs are retried once before persistent failures are reported as missing.Source: §D.4 · p. 17

E.1 SENSITIVITY TO THE VISUAL TEACHER

  • With the selector setup held fixed, DINOv2, CLIP, and SigLIP teachers all improve consistency, with DINOv2 performing best.The additional ablations use frozen Self-Forcing, four historical frames, and four prospective queries unless otherwise specified.Source: §E.1 · p. 17, Table 12 · p. 17, E ADDITIONAL ABLATIONS · p. 17

E.2 RANKING OBJECTIVE

  • Combining listwise distribution matching and margin-based pairwise ordering constraints yields better consistency than either supervision alone.Listwise supervision is stronger than pairwise supervision when used alone.Source: §E.2 · p. 18, Table 13 · p. 18

E.3 LENGTH OF THE REALIZED CONTINUATION

  • Consistency improves substantially through H = 4 continuation observations and then largely saturates, while longer horizons add limited benefit.The continuation horizon changes teacher supervision, not deployed inputs or computation.Source: §E.3 · p. 18, Table 14 · p. 18

E.4 AGGREGATION OF PROSPECTIVE QUERIES

  • Smooth-maximum query-score aggregation gives the best consistency among mean, hard-maximum, and smooth-maximum pooling.It lets one strong prospective match dominate while retaining contributions from other queries.Source: §E.4 · p. 18

E.5 RELATION BETWEEN FUTURE RELEVANCE AND GENERATION UTILITY

The diagnostic tests whether future-grounded frame rankings track downstream generation utility, rather than only visual correspondence with a realized continuation. It finds that the teacher ranking aligns more closely with utility than current-context similarity, and that FRAMEMORROW recovers much of this association without future access at inference.

Utility measurement

  • Candidate utility is assessed by comparing a native-context baseline continuation with continuations supplied the candidate frame, holding context, seed, sampling settings, and backbone fixed.The long-range consistency evaluator measures preservation of subjects, objects, and scene appearance from preceding context.Source: §E.5 · pp. 18–19

Correlation analysis

  • Rank correspondence is computed among historical candidates at each rollout step, then averaged within trajectories and across 30 held-out trajectories.The analysis uses 120 valid steps and eight candidates per step, with trajectory-level bootstrap confidence intervals.Source: §E.5 · p. 19

Diagnostic setup

  • The diagnostic compares current-context similarity, a future-grounded teacher ranking, and FRAMEMORROW’s predicted ranking against per-frame generation utility.The teacher uses held-out realized continuations for this analysis, while FRAMEMORROW does not observe future content at inference.Source: §E.5 · pp. 18–19

Finding

  • Current-context similarity has weak correspondence with generation utility, while the teacher ranking aligns more closely and FRAMEMORROW recovers much of that association without future access at inference.These results support continuation-based visual correspondence as a training proxy for downstream usefulness.Source: §E.5 · p. 19, Table 16 · p. 19

F ADDITIONAL QUALITATIVE RESULTS

The section presents visual comparisons of native and FRAMEMORROW-augmented generators under matched conditions.

F ADDITIONAL QUALITATIVE RESULTS

  • Figures 7–9 compare native and FRAMEMORROW-augmented Self-Forcing, LongLive, and Causal Forcing under matched generation conditions to illustrate long-horizon visual consistency.Source: F ADDITIONAL QUALITATIVE RESULTS · p. 19

G USER STUDY

A blinded pairwise study found FrameMorrow preferred over native models for consistency and visual quality across three video-generation backbones. Qualitative comparisons also show improved preservation of subjects, objects, and scene details during long-horizon generation.

G USER STUDY

  • Ten blinded raters judged 30 randomly sampled native–FrameMorrow video pairs per backbone, assessing long-range consistency and overall visual quality.The study covered Self-Forcing, LongLive, and Causal Forcing; each criterion received 900 judgments.Source: G USER STUDY · pp. 19–20
  • Qualitative comparisons show improved subject appearance and visual-detail preservation with FrameMorrow for Self-Forcing, LongLive, and Causal Forcing.The comparisons depict better scene appearance for LongLive and more consistent subjects, objects, and scene details for Causal Forcing over extended generation.Source: Figure 7 · p. 20, Figure 8 · p. 20, Figure 9 · p. 21
  • 62.7% consistency preferences and 54.0% quality preferences favored FrameMorrow, versus 16.7% and 17.0% for native models, with 20.7% and 29.0% ties, respectively.Figure 10 reports preferences averaged across the three backbones.Source: G USER STUDY · pp. 20–21, Figure 10 · p. 21

Related papers

Loading 2609.38839v1…