PaperlayerView exact revision on arXiv

Source-linked AI summary

Seer: Language Instructed Video Prediction with Latent Diffusion Models

Xianfan Gu, Chuan Wen, Weirui Ye, Jiaming Song, Yang Gao

arXiv:2303.14897v4cs.CV

TL;DR

Text-conditioned video prediction must generate future frames grounded in both initial frames and language instructions. Seer inflates pretrained text-to-image diffusion models and decomposes instructions into frame-aligned guidance; it reports strong benchmark performance and human preference.

  • Problem

    Text-conditioned video prediction requires models to ground language instructions in initial frames while predicting temporally detailed future movement.

    Source: §1 · p. 1

  • Method

    Seer inflates pretrained text-to-image diffusion models along time and uses a Frame Sequential Text Decomposer to produce fine-grained guidance for each frame.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    Seer achieves the best baseline results across SSv2, BridgeData, and Epic100, including FVD 112.9 and KVD 0.12 on SSv2; human evaluation preference is at least 72.2%.

    Source: §5.4 · p. 8

  • Takeaways & Limitations

    Seer generates high-quality videos by fine-tuning only the SAWT-Attn and FSText Decomposer, reducing data and computation costs.

    Source: §6 · pp. 9–10

Abstract

from arXiv · show

Imagining the future trajectory is the key for robots to make sound planning and successfully reach their goals. Therefore, text-conditioned video prediction (TVP) is an essential task to facilitate general robot policy learning. To tackle this task and empower robots with the ability to foresee the future, we propose a sample and computation-efficient model, named \textbf{Seer}, by inflating the pretrained text-to-image (T2I) stable diffusion models along the temporal axis. We enhance the U-Net and language conditioning model by incorporating computation-efficient spatial-temporal attention. Furthermore, we introduce a novel Frame Sequential Text Decomposer module that dissects a sentence's global instruction into temporally aligned sub-instructions, ensuring precise integration into each frame of generation. Our framework allows us to effectively leverage the extensive prior knowledge embedded in pretrained T2I models across the frames. With the adaptable-designed architecture, Seer makes it possible to generate high-fidelity, coherent, and instruction-aligned video frames by fine-tuning a few layers on a small amount of data. The experimental results on Something Something V2 (SSv2), Bridgedata and EpicKitchens-100 datasets demonstrate our superior video prediction performance with around 480-GPU hours versus CogVideo with over 12,480-GPU hours: achieving the 31% FVD improvement compared to the current SOTA model on SSv2 and 83.7% average preference in the human evaluation.

1 INTRODUCTION

Seer targets TVP’s need to predict temporally coherent, instruction-aligned future frames from reference frames and language. It adapts pretrained text-to-image models and reports improved benchmark performance with reduced training costs.

1 INTRODUCTION

  • TVP must ground language in initial frames and model temporal movement, unlike text-to-video generation that does not explicitly condition on initial frames.The introduction identifies large labeled datasets and compute, low frame fidelity, and a lack of frame-level instruction as performance limitations.Source: §1 · pp. 1–2
  • Seer inflates pretrained T2I models along the temporal axis and decomposes global instructions into fine-grained, time-aligned guidance for each frame.Its Inflated 3D U-Net models spatial and temporal dependencies, while the Frame Sequential Text Decomposer uses causal attention.Source: §1 · p. 2
  • FVD falls from 163 to 113 on Something-Something V2 compared with TAV.The introduction also reports human preference rates above 70% over TAV and around 90% over TATS and MCVD.Source: §1 · pp. 2–3
  • The experiments report 120 hours on 4 RTX 3090 GPUs and human preference above 70% over TAV and around 90% over TATS and MCVD.The paper presents these results as evidence of training efficiency and improved generation quality.Source: §1 · p. 3

2 RELATED WORK

The related work traces how pretrained text-to-image models and diffusion methods inform video generation. Seer adapts this line of work with autoregressive attention and temporally decomposed language guidance.

2.1 TEXT-TO-IMAGE GENERATION

  • Latent diffusion reduces denoising computation by operating in latent space, and Seer uses pretrained LDM language-vision knowledge while extending the model along time.The subsection reviews T2I advances including diffusion, classifier-free guidance, and VAE-based latent diffusion.Source: §2.1 · p. 3

2.2 TEXT-TO-VIDEO GENERATION

  • Text-to-video generation remains constrained by text-video data and compute; prior approaches include 3D sparse-attention, nearby-attention, and pretrained hierarchical models.The subsection names GODIVA, NÜWA, and CogVideo as examples.Source: §2.2 · p. 3
  • Unlike the cited T2V approaches, Seer applies autoregressive attention across spatial and temporal spaces and decomposes language into fine-grained sub-instructions for task-level prediction.The passage contrasts Seer with Make-A-Video, MagicVideo, Tune-A-Video, and Imagen Video, which transfer 2D diffusion models to 3D using temporal modules.Source: §2.2 · p. 3

3 PRELIMINARIES

The preliminaries describe diffusion as iterative noising and denoising, then introduce classifier-free guidance for conditional generation. They also explain latent diffusion’s use of a VAE-based latent representation and condition encoder.

3 PRELIMINARIES

  • Diffusion models approximate a data distribution through a forward process that adds noise and a reverse process that denoises samples.The forward process forms noisy samples from data and accumulated noise; the reverse process estimates prior samples from noisy ones.Source: §3 · pp. 3–4
  • Classifier-free guidance combines conditional and unconditional noise predictions using a guidance scale.The unconditional prediction replaces the condition with a null token.Source: §3 · p. 4
  • Latent diffusion uses a VAE-based latent representation for diffusion and a domain-specific encoder to project the generation condition.The passage motivates latent diffusion as a way to reduce the higher computation costs of video diffusion.Source: §3 · p. 4

4 METHODOLOGY

Seer adapts pretrained text-to-image diffusion models for video prediction, combining spatial-temporal modeling with frame-aligned language guidance. Its method uses an inflated 3D U-Net and a Frame Sequential Text Decomposer to address limited data and computation, frame fidelity, and coarse instruction guidance.

4.1 OVERVIEW OF SEER

  • Seer adapts pretrained text-to-image diffusion models for video prediction and decomposes global instructions into temporally aligned sub-instructions for more precise guidance.The paper identifies limited labeled text-video data and computation, low frame fidelity, and insufficient fine-grained frame instructions as challenges.Source: §4.1 · p. 4, §4 · p. 4
  • The method uses reference-frame and language pathways: spatial-temporal layers propagate reference context, while FSText generates multi-timestep sub-instructions injected into each frame’s latent space.Only the spatial-temporal layers and FSText module are trained from scratch; the remaining pretrained modules are frozen.Source: §4.1 · pp. 4–5

4.2 DATA & COMPUTATION-EFFICIENT 3D NETWORK

  • The inflated 3D U-Net extends pretrained 2D modules with 3D spatial layers and autoregressive local-window attention using a causal mask across spatial-temporal dimensions.The causal mask constrains pixel propagation from future temporal-spatial positions.Source: §4.2 · p. 5, Figure 3 · p. 5

4.3 FRAME SEQUENTIAL TEXT DECOMPOSER

  • FSText applies text-sequential attention, cross-attention from learnable sub-instruction tokens to global text, then directed temporal attention to produce frame-specific guidance.The learnable tokens have shape (n, l, C), with n denoting the number of frames.Source: §4.3 · pp. 5–6, Figure 5 · p. 6
  • Frame-individual cross-attention pairs each frame’s visual latent vectors with its own sub-instruction embeddings, then concatenates the per-frame results.The method initializes the FSText decomposer as an identity function to keep sub-instruction embeddings close to CLIP text embeddings.Source: §4.3 · p. 6, Figure 4 · p. 6

5 EXPERIMENTS

Seer is evaluated on three text-video datasets against video-prediction baselines, with human and qualitative comparisons, component ablations, instruction-guidance analysis, and zero-shot testing.

5 EXPERIMENTS

  • Experiments use SSv2, BridgeData, and Epic100, with videos downsampled to 12 frames for SSv2 and Epic100 and 16 for BridgeData; training schedules and baselines vary by dataset and method.The implementation initializes from Stable Diffusion-v1.5 and uses DDIM sampling with 30 timesteps; evaluation includes different reference-frame counts across datasets.Source: §5.1 · pp. 6–7, §5.2 · p. 7, §5.3 · p. 7

5.4 MAIN RESULTS

  • At least 72.2% of human-evaluation comparisons preferred Seer for semantic content and video fidelity.The evaluation covered text-conditioned video manipulation on SSv2.Source: Figure 7 · p. 7, §5.4 · p. 8
  • 112.9 FVD and 0.12 KVD on SSv2, 246.3 FVD and 0.55 KVD on BridgeData, and 271.4 FVD on Epic100 were Seer’s best baseline results.Table 1 reports the three-dataset comparison.Source: Table 1 · p. 8, §5.4 · p. 8
  • Qualitative comparisons show stronger temporal handling and text alignment, including coherent camera turning, hidden-object prediction, and better BridgeData behavior alignment.The figures compare SSv2 prompt variants and BridgeData predictions.Source: Figure 6 · p. 7, §5.4 · p. 8, Figure 8 · p. 8

5.5 ABLATION STUDY

  • Scaled autoregressive window attention had the lowest FVD and KVD among tested attention variants, while directed attention also beat bidirectional attention.The authors attribute the latter difference to the sequential-generation inductive bias.Source: §5.5 · pp. 8–9, Table 2 · p. 8
  • Identity initialization improved FSText prediction quality, and the default setting of fine-tuning FSText plus temporal attention outperformed the tested alternatives.Further fine-tuning spatial-cross attention alongside temporal attention substantially reduced performance.Source: §5.5 · p. 9, Table 3 · p. 9, Table 4 · p. 9

5.6 VISUAL ANALYSIS OF INSTRUCTION EMBEDDING

  • Frame-wise sub-instructions guide motion at each timestep; cloning a later sub-instruction led to incoherent transitions, while an earlier clone guided motion only to an intermediate state.The comparison tests clones from the third and twelfth frames against default frame-wise guidance.Source: §5.6 · p. 9, Figure 9 · p. 9

5.7 ZERO-SHOT EVALUATION

  • After fine-tuning on SSv2, Seer outperformed VideoFusion on both FVD and KVD in zero-shot evaluation on EGO4D.The authors describe this as evidence of generalizability to the unseen dataset.Source: §5.7 · p. 9, Table 5 · p. 9

6 CONCLUSION

Seer inflates pretrained text-to-image Stable Diffusion into a video model and adds frame-sequential text decomposition. It fine-tunes only selected layers and reports superior performance over recent models.

6 CONCLUSION

  • Seer inflates pretrained text-to-image Stable Diffusion along the temporal axis and adds a Frame Sequential Text Decomposer.The architecture is presented as sample- and computation-efficient for text-conditioned video prediction.Source: §6 · p. 9
  • The method fine-tunes only SAWT-Attn and FSText Decomposer layers, reducing data and computation costs.Source: §6 · p. 10
  • The experiments report superior performance over recent models.Source: §6 · p. 10

7 REPRODUCIBILITY STATEMENT

The reproducibility material points to the main training and dataset sections, details baseline protocols, and reports an auxiliary UCF-101 comparison whose benchmark and model constraints differ from the paper’s TVP focus.

7 REPRODUCIBILITY STATEMENT

  • The reproducibility statement directs readers to Sections 5.1–5.2 and Appendices B.1 and C for implementation, experiment, and hyperparameter details.Source: §7 · p. 10

B.1 IMPLEMENTATION DETAILS OF BASELINES

  • Baseline protocols specify different training resolutions and step counts across methods, including 200k steps for Tune-A-Video and VideoFusion and 300k for several others.Source: §B.1 · p. 15

B.2 EVALUATION DETAILS AND RESULTS OF UCF-101

  • UCF-101 evaluates class-conditioned short-horizon clips rather than task-level TVP, so the authors include it for comparison despite its imperfect fit.Source: §B.2 · p. 15
  • The UCF-101 evaluation uses 16-frame videos conditioned on five reference frames and reports FVD and Inception Score.FVD uses 2,048 validation samples and IS uses 100k.Source: §B.2 · p. 15
  • On UCF-101, Seer outperformed CogVideo and MagicVideo but fell short of Make-A-Video, which used unlabelled video pretraining.The authors state that addressing Seer’s lack of video pretraining and its unfine-tuned 2D VAE is outside this paper’s TVP scope.Source: §B.2 · p. 15

B.3 EVALUATION RESULTS OF SAMPLING STEPS

The appendix examines DDIM sampling, longer-sequence behavior, and computational efficiency. Seer retains advantages over Tune-A-Video at low sampling steps, while frozen spatial layers reduce memory use and training time.

B.3 EVALUATION RESULTS OF SAMPLING STEPS

  • Increasing DDIM steps notably improved longer-video quality from 10 to 30 steps, but gains diminished beyond 30.The evaluation used a 12-frame SSv2-fine-tuned model for both 12- and 16-frame outputs.Source: §B.3 · pp. 15–16, Figure 10 · p. 16
  • At 10 DDIM steps, Seer achieved 280.7 FVD and 0.73 KVD, versus Tune-A-Video’s 419.3 FVD and 1.5 KVD.Seer also consistently outperformed Tune-A-Video across the tested 10–50-step range on both metrics.Source: §B.3 · p. 16, Figure 11 · p. 17
  • A 12-frame model generated task motion in 16-frame outputs, but appearance and temporal consistency declined as sequence length increased.The authors suggest training a dedicated 16-frame Seer model to improve those outputs.Source: §B.3 · p. 16, Figure 12 · p. 17

B.4 COMPUTATION EFFICIENCY OF SEER

  • Freezing Seer’s 2D spatial layers used 24.7GB of GPU memory and 3.10 sec/iter, versus 39.2GB and 6.89 sec/iter without freezing and 45.0GB and 7.63 sec/iter for VideoFusion.The memory comparison is reported in Table 8 and the training-time comparison in Table 9.Source: Table 8 · p. 17, §B.4 · pp. 17–18, Table 9 · p. 17
  • Scaled autoregressive window attention reduced computational overhead relative to plain autoregressive spatial-temporal attention while retaining better generation quality than bidirectional and directed variants.The appendix reports temporal-layer computational costs in Table 7.Source: Table 7 · p. 17, §B.4 · p. 17

B.5 ADDITIONAL ABLATION RESULTS

The additional ablations favor Seer’s default temporal-attention and FSText design across component, depth, fine-tuning, and dataset comparisons. In simulated policy learning, videos generated by fine-tuned Seer provide scalable data of comparable quality to the in-domain videos.

B.5 ADDITIONAL ABLATION RESULTS

  • The default 8-layer FSText Decomposer outperforms shallower variants on FVD, despite a marginal KVD advantage for the 4-layer model.The authors adopt the 8-layer version for its FVD performance and text-video alignment robustness.Source: §B.5 · p. 18
  • Combining temporal and text-sequential attention outperforms replacing either component with cross-attention in the FSText ablation.The authors associate the two components with capturing text context and modeling temporal dependencies, respectively.Source: §B.5 · p. 18
  • BridgeData ablations reproduce the improvement patterns from SSv2 fine-tuning and temporal-layer studies, supporting consistency across the two datasets.The comparisons are reported in Tables 12 and 13 and are described as mirroring SSv2 Tables 2 and 4.Source: §B.5 · p. 18
  • The default “temp+FSText” setting preserves stronger temporal consistency and text-based motion than the other fine-tuning settings in qualitative comparisons.Figure 13 presents these fine-tuning-setting comparisons.Source: §B.5 · p. 18, Figure 13 · p. 19

B.6 EVALUATION OF POLICY LEARNING ON SEER

  • Fine-tuning Seer on 10 in-domain videos per task and generating 1,000 videos per task yielded policy-learning data of comparable quality to using the 10 videos directly.The comparison is against a policy distilled directly from the 10 in-domain videos; Table 14 reports success rates.Source: Table 14 · p. 19, §B.6 · p. 19

C.1 FINE-TUNING AND SAMPLING

  • Table 15 lists Seer’s fine-tuning and sampling hyperparameters and hardware information.The section describes these as implementation details.Source: §C.1 · p. 20

C.2 ARCHITECTURE INFORMATION

This section provides reference tables for Seer’s fine-tuning and inference settings and the architectures of its 3D U-Net and FSText Decomposer.

C.2 ARCHITECTURE INFORMATION

  • Table 16 lists the 3D U-Net hyperparameters.The section also identifies the U-Net hyperparameter table.Source: §C.2 · p. 20, Table 16 · p. 20
  • Table 17 lists the FSText Decomposer hyperparameters.The section also identifies the decomposer hyperparameter table.Source: §C.2 · p. 20, Table 17 · p. 21
  • Table 15 lists hyperparameters and details for fine-tuning and inference.The table is identified as a fine-tuning/inference reference.Source: Table 15 · p. 20

D.1 ADDITIONAL QUALITATIVE RESULTS

The appendix adds qualitative comparisons of Seer’s instruction alignment, transfer to new objects and robot-simulation datasets, and longer-horizon generation. It also documents failure cases involving environmental continuity and scene transitions.

D.1 ADDITIONAL QUALITATIVE RESULTS

  • Additional visualizations show Seer’s text-conditioned predictions on SSv2, BridgeData, and Epic-Kitchens-100, including prediction and manipulation examples.Figure 14 adds 12-frame SSv2 predictions conditioned on the first reference frame.Source: Figure 14 · p. 21, §D.1 · p. 21

D.2 GENERALIZABILITY OF SEER ON DOWNSTREAM TASKS

  • On BridgeData, the default fine-tuning setting recognizes the unseen “black mug” and preserves background consistency with higher fidelity than the compared LORA setting.Both settings recognize the object; Figure 15 compares the visual results.Source: Figure 15 · p. 21, §D.2 · p. 21

D.4 ADDITIONAL VISUALIZATION OF VIDEO MANIPULATION WITH UNSEEN OBJECTS AND ZERO-SHOT VIDEO MANIPULATION

  • After mixed SSv2–BridgeData fine-tuning, Seer recognizes unseen BridgeData objects and performs a pick-up motion, but interaction can be limited for an unseen action.On zero-shot EGO4D, it identifies objects and acts from prior human-activity knowledge, yet may fail to ground the action in the reference scene.Source: §D.4 · p. 22, Figure 17 · p. 23

D.3 ADDITIONAL VISUALIZATION OF SUB-INSTRUCTION EMBEDDING

  • Frame-specific sub-instructions guide motion through time: a third-frame embedding stalls the example motion partway, while a twelfth-frame embedding disrupts scene continuity.The comparison illustrates that the sub-instructions track the temporal sequence of the global instruction.Source: §D.3 · p. 22, Figure 16 · p. 23

D.5 VISUALIZATION OF ROBOT SIMULATION ENVIRONMENT

  • After mixed-dataset fine-tuning, Seer generates coherent, instruction-aligned motion on Meta-World, CLIPort, and RLBench simulation data.The examples include locating small target objects in CLIPort and multi-step tasks in RLBench.Source: Figure 18 · p. 24, §D.5 · p. 24

D.6 LONG VIDEO PREDICTION

  • For long video generation with a 12-frame model, repeating the first clip’s sub-instructions maintains coherent motion, whereas interpolating them can introduce noise.The clips are generated sequentially, with the first clip’s last two frames conditioning the next; the reported examples use the 12-frame model.Source: §D.6 · pp. 24–25, Figure 19 · p. 25

D.7 FAILURE CASE

  • Failure examples show correct task-motion and object predictions alongside appearance mismatches, while Epic-Kitchens scene transitions can prompt camera-motion and novel-view predictions.In a wine-pouring example, Seer generates a glass based on learned knowledge but overlooks the transition distribution from the reference environment.Source: §D.7 · p. 26, Figure 20 · p. 26

E HUMAN EVALUATION DETAILS

The human evaluation used paired video choices to assess instruction consistency and fidelity, with responses gathered for comparisons against three baselines. Its protocol included 54 evaluators and 99 SSv2 validation clips.

E HUMAN EVALUATION DETAILS

  • The study used 99 SSv2 validation clips and 54 anonymous evaluators, each receiving 20 randomly selected questions.Each single-choice question showed a ground-truth reference, a manually modified instruction, and predictions from Seer and a baseline.Source: E HUMAN EVALUATION DETAILS · p. 27
  • The evaluation collected 342 Seer–TATS responses, 363 Seer–Tune-A-Video responses, and 357 Seer–MCVD responses.The main-paper Figure 7 results were calculated from these questionnaires.Source: E HUMAN EVALUATION DETAILS · p. 27
  • Evaluators chose which prediction better matched the instruction and had higher fidelity, with text-based motion prioritized over visual fidelity.Figure 21 shows an example questionnaire.Source: Figure 21 · p. 27, E HUMAN EVALUATION DETAILS · p. 27
Loading 2303.14897v4…