PaperlayerView exact revision on arXiv

Source-linked AI summary

First Frame Is the Place to Go for Video Content Customization

Jingxi Chen, Zongxia Li, Zhichao Liu, Guangyao Shi, Xiyang Wu, Fuxiao Liu, Cornelia Fermuller, Brandon Y. Feng, Yiannis Aloimonos

arXiv:2511.15700v2cs.CV

TL;DR

This work asks whether pretrained video generation models can combine content from multiple reference images without architectural changes or large-scale video-training datasets. The method uses the first frame as a conceptual memory buffer and applies few-shot LoRA adaptation with 20–50 examples to invoke subject fusion and scene transitions. The authors report that their 50-video adaptation outperforms VACE and SkyReels-A2 across a wide range of scenarios.

  • Problem

    This work asks whether pretrained video generation models can combine content from multiple reference images without architectural changes or large-scale video-training datasets.

    Source: §1 · p. 2

  • Method

    The method uses the first frame as a conceptual memory buffer and applies few-shot LoRA adaptation with 20–50 examples to invoke subject fusion and scene transitions.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    The authors report that their 50-video adaptation outperforms VACE and SkyReels-A2 across a wide range of scenarios.

    Source: §4.2 · p. 5

  • Takeaways & Limitations

    The first frame can function as a conceptual memory buffer for storing and fusing disjoint reference subjects for downstream video generation.

    Source: §6 · p. 8

    Performance is reported to work well up to four subjects plus a reference scene; beyond five references, identity preservation and prompt-based selection degrade.

    Source: §5 · p. 8

Abstract

from arXiv · show

What role does the first frame play in video generation models? Traditionally, it's viewed as the spatial-temporal starting point of a video, merely a seed for subsequent animation. In this work, we reveal a fundamentally different perspective: video models implicitly treat the first frame as a conceptual memory buffer that stores visual entities for later reuse during generation. Leveraging this insight, we show that it's possible to achieve robust and generalized video content customization in diverse scenarios, using only 20-50 training examples without architectural changes or large-scale finetuning. This unveils a powerful, overlooked capability of video generation models for reference-based video customization.

1. Introduction

Multi-reference video customization can provide richer composition than single-image I2V, but existing approaches often require architectural changes or large-scale fine-tuning that can reduce generalization. The paper proposes using the first frame as a conceptual memory buffer and a few-shot adaptation to invoke that capability.

1. Introduction

  • Existing multi-reference approaches modify architectures or use large task-specific datasets, while fine-tuning can degrade performance and generalization through overfitting.Because customization datasets are less diverse and lower quality than pre-training data, adapted models may forget broad generative priors.Source: §1 · p. 2
  • The paper asks whether pre-trained video models can incorporate multiple references without architectural changes or large-scale video training datasets.Multi-reference generation offers richer and more flexible customization than single-image I2V, which constrains spatial diversity and content composition.Source: §1 · p. 2
  • The paper identifies the first frame as a conceptual memory buffer that embeds visual concepts from multiple references for later fusion through scene transitions.This reframes the first frame from only a spatiotemporal start point into a store for visual concepts used during generation.Source: §1 · p. 2
  • FFGo uses VLM-curated data and few-shot LoRA adaptation with 20–50 examples to invoke multi-reference customization without changing model architecture or compromising pre-trained generative priors.The paper evaluates the add-on across filmmaking, multi-object interaction, several simulation domains, and robotic manipulation.Source: §1 · p. 2

2. Related Work

Prior customization methods commonly rely on architectural changes and specialized training, while pre-trained generative models can exhibit broader innate capabilities. The paper connects these capabilities to reference-based customization and notes both the potential and practical challenges of subject mixing through scene transitions.

Video Generation and Video Content Customization

  • Video customization approaches typically add visual inputs through architectural modifications and specialized large-scale training, which can compromise efficiency, compatibility, and broad pre-training knowledge.The cited work contrasts the diversity and quality of pre-training data with limited-domain customization fine-tuning.Source: §2 · pp. 2–3
  • Video models can mix subjects from a mixed-subject first frame through scene transitions, but prompt selection is manual and transitions or object identities can be unstable.Figure 2 contrasts compositions generated with and without a carefully chosen transition phrase.Source: Figure 2 · p. 3

The Innate Abilities of Pre-trained Generative Models

  • Pre-trained generative models can exhibit under-explored general-purpose abilities beyond their original design goals, including frame-level perception in I2V models.Prior examples include few-shot LoRA adaptation for coherent grid-aligned image generation and video-model edge detection, segmentation, and super-resolution.Source: §2 · p. 3

Vision-language models for multimodal data curation

  • Unified VLMs process images, videos, and text and support instruction-following image editing and video understanding, making them useful for curating training data.The passage lists Gemini-2.5-Pro, Qwen2.5-Omni, and GPT-4o as examples.Source: §2 · p. 3

3. Proposed Approach

FFGo curates paired training data, applies few-shot LoRA adaptation to invoke subject fusion and scene transitions, then removes the transition frames to deliver clean customized videos.

3.1. Pipeline Overview

  • FFGo combines VLM-based dataset curation, few-shot LoRA adaptation, and clean customized video inference.The pipeline uses curated paired data to invoke the model’s innate fusion and transition abilities for generalized multi-reference generation.Source: §3.1 · p. 3, Figure 3 · p. 4

3.2. Dataset Curation

  • The curation process selects videos with clear interactions or combinations, extracts named elements and backgrounds, and pairs the compositions with prompts describing appearance and interaction.From 2,000 gathered videos, the authors select 50 annotated examples; VLMs extract elements and backgrounds, and generate video-descriptive prompts.Source: §3.2 · pp. 4–5

3.3. Few-shot LoRA Adaptation

  • With 20–50 examples, LoRA adaptation trains a pre-trained I2V model to use a subject-mixed first frame and transition-marked prompt for subject mixing and scene transitions.The adapted output uses a learnable low-rank weight update and produces generated content after temporal-compression frames.Source: §3.3 · p. 5

3.4. Clean Customized Video Inference

  • During inference, users can cut off the first Fc=4 frames of an 81-frame Wan2.2 video, leaving Fg=77 frames of clean customized content after subject mixing and a temporal scene transition.Source: §3.4 · p. 5

4. Experimental Results

The experiments compare FFGo with the Wan2.2-I2V-A14B base model and two Wan-based customization baselines across diverse customization scenarios. Qualitative comparisons and a user study report stronger identity preservation and customization performance with lightweight adaptation.

4.1. Datasets and Implementation Details

  • The curated training set contains 50 videos across human-object, human-human, element-insertion, and robot-manipulation categories, with 81 frames per video.The videos were selected from a pool of 2,000; LoRA adaptation uses a unique transition phrase to trigger subject selection and scene transition.Source: §4.1 · p. 5

4.2. Evaluation Strategy

  • The evaluation uses 50 reference sets across diverse applications, with up to five reference subjects; comparisons include Wan2.2-I2V-A14B, VACE, and SkyReels-A2.The authors describe the test set as covering broader real-world customization scenarios and more subjects than prior works.Source: §4.2 · p. 5

4.4. Quantitative Comparison

  • Over 80% of users selected FFGo as their top choice, and it outperformed all baselines on ranking, object identity, scene identity, and overall quality.The user study reports 200 annotations from 40 users; FFGo used a lightweight adaptation.Source: Table 1 · p. 6, §4.4 · p. 8

4.3. Qualitative Comparison

  • Unlike the base model, which often animates elements independently and loses referred objects after transitions, the adapted model preserves object identities and generates coherent transitions.In rare cases when the base model succeeds at both, FFGo’s output closely resembles its output, indicating retained pre-trained generative capabilities.Source: §4.3 · p. 6, Figure 4 · p. 6
  • Across diverse scenarios, FFGo trained on 50 examples outperformed VACE and SkyReels-A2, which were trained on millions of videos for specific customization tasks.The authors attribute baseline generalization limits to overfitting; FFGo activates the base model’s innate capabilities while preserving its general knowledge.Source: §4.3 · p. 6

5. Limitations

The method’s practical capacity is limited by the number of references: more subjects reduce the resolution available to each and make prompt-based selection harder. It performs well with up to four subjects plus a reference, while performance degrades beyond five references total.

5. Limitations

  • More reference subjects reduce per-subject resolution and make it harder to target specific objects using text prompts.The method performs well with up to four subjects plus a reference scene; beyond five references total, identity preservation and prompt-based selection degrade.Source: §5 · p. 8
  • Multiple start frames are proposed as a possible extension of the conceptual memory buffer for higher-capacity reference encoding.The authors describe this as a future engineering enhancement, not a demonstrated result.Source: §5 · p. 8

6. Conclusions

The paper reframes the first frame as a conceptual memory buffer for storing and fusing reference subjects during video generation. It presents a lightweight few-shot adaptation that achieves strong customization performance across real-world scenarios and aligns with user preferences.

6. Conclusions

  • The first frame can store and fuse disjoint reference subjects for downstream video generation.The paper presents this as a departure from treating the first frame only as the animation’s spatiotemporal starting point.Source: §6 · p. 8
  • A lightweight add-on adaptation turns a base video model into a customization system without architectural changes or large-scale finetuning.The paper reports strong performance across real-world scenarios and alignment with user preferences in a comprehensive user study.Source: §6 · p. 8

A. Video Results

The section points readers to video examples and describes a comparison with a two-stage image-composition-plus-animation baseline. The authors report that their method preserves temporal control through text prompts, unlike a pipeline whose fixed first frame dictates the spatial layout.

A. Video Results

B. Comparison with Two-Stage Baselines

  • The method is end-to-end and preserves fine-grained temporal control, including prompts for an entity to join later.The described two-stage baseline first composes an image layout and then animates it with an I2V model; its fixed first frame limits temporal flexibility.Source: B. Comparison with Two-Stage Baselines · p. 12
  • The two-stage pipeline’s fixed first frame dictates the entire spatial layout, limiting temporal flexibility.The authors contrast this with prompts such as “a shark joins the party later,” which their method faithfully realizes.Source: B. Comparison with Two-Stage Baselines · p. 12

C.1. Training Dataset Curation Details

The training and test curation procedures build composite reference canvases from foreground cut-outs and backgrounds. Training clips are standardized and paired with captions, while the test set uses curated object-background pairs and refined prompts.

C.1. Training Dataset Curation Details

  • The training corpus draws on 2,205 candidate clips from HOIGen-1M, Veo 3 demonstrations, and licensed short videos, filtered to 50 high-quality examples.The retained examples cover human–object interaction, human–human interaction, element insertion, and robot manipulation.Source: §C.1 · p. 12
  • Training clips are standardized to 81 frames, and foreground entities are manually tagged after each clip’s first frame is extracted.The tagged entities are used in subsequent object extraction and background cleanup.Source: §C.1 · p. 12
  • Foreground cut-outs and cleaned backgrounds form the compositional basis of each training first frame.A 1280×720 canvas places the foreground cut-outs on the left and the background on the right, serving as conditioning input and the initial frame.Source: §C.1 · p. 13
  • Training captions are generated from object cut-outs, clean background plates, and full 81-frame videos using a structured prompt template.Source: §C.1 · p. 13

C.2. Test Set Curation

  • The test set pairs manually segmented foreground objects with backgrounds on 1280×720 canvases, then refines prompts to produce 50 image-prompt pairs.Source: §C.2 · p. 13

D. Details about User Study

The paper reports that its model can handle some unseen first-frame layouts and maintain visual consistency when reference inputs come from different sources. The supplied passages also describe the user-study procedure and model training setup.

D. Details about User Study

D.1. User Study Platform

  • The study recruited 40 participants with video-annotation experience and fluent English, who each annotated five sets for about 15 minutes and received $5.50.Participants were recruited through Prolific.Source: §D.1 · pp. 13–14

D.2. User Interface Details

  • Participants saw four videos in randomized order, ranked overall quality, then rated object identity, scene identity, and overall quality on a five-point scale.The interface also collected global rankings and aspect-wise ratings.Source: §D.2 · p. 14, Annotation Task Instructions · p. 15

Annotation Task Instructions

  • Each annotation set presented a five-second silent video task with a prompt, a reference image split between foreground objects and background, and four generated outputs.Source: Annotation Task Instructions · p. 15

E. More Training and Inference Details

  • LoRA modules of rank 128 were trained on 81-frame videos at 1344 × 768; inference used 81 frames at 1280 × 720.The training-data categories were human–object interaction (60%), human–human interaction (14%), element insertion (20%), and robot manipulation (6%).Source: Figure 11 · p. 13, E. More Training and Inference Details · p. 15

F. Generalization to First-Frame Layouts

  • The model generalizes to three unseen first-frame layouts in some cases, suggesting it interprets first-frame context rather than relying only on the training layout.The training layout places cut-outs on the left and the background on the right.Source: F. Generalization to First-Frame Layouts · p. 15

G. Visual Consistency Across Generated Videos from Different Reference Sources

  • References drawn from different sources can still yield visual consistency, including correctly rendered hand-and-bottle shadows on a teddy bear.The paper attributes consistency to the pre-trained models’ learned priors.Source: G. Visual Consistency Across Generated… · p. 15

H. Automatic Quantitative Metrics

The paper reports automatic concept and VBench metrics for comparison with baselines, and explains how a unique transition phrase works with LoRA training to trigger scene cuts and reference fusion.

H. Automatic Quantitative Metrics

  • The evaluation reports CLIP-I, DINO-I, CLIP-cap, Motion Smoothness, Dynamic Degree, Aesthetic Quality, and Imaging Quality to compare with baselines.The passage says these metrics further validate the method, but supplies no metric values.Source: H. Automatic Quantitative Metrics · p. 15
  • Figures 16 and 17 address generalization to first-frame spatial layouts and references from different sources, respectively.Source: Figure 16 · p. 16, Figure 17 · p. 16

I. Explanation of the Transition Phrase

  • A unique transition phrase paired with LoRA training teaches the base model to invoke latent scene-cut and reference-fusion abilities at inference.The trigger can be arbitrary if it is unique; its use differs from DreamBooth’s trigger phrases.Source: §I · p. 16
Loading 2511.15700v2…