PaperlayerView exact revision on arXiv

Source-linked AI summary

Warp-as-History: Generalizable Camera-Controlled Video Generation from One Training Video

Yifan Wang, Tong He

arXiv:2605.15182v1cs.CV

TL;DR

Camera-controlled video generation still often requires camera-specific training, test-time optimization, or extra guidance. Warp-as-History instead routes camera-warped pseudo-history through a frozen model’s native history pathway, while one-video LoRA finetuning improves generalization and camera adherence across diverse datasets.

  • Problem

    Existing camera-control methods often require large-scale camera-annotated training, while training-free alternatives may require test-time optimization or extra guidance.

  • Method

    Warp-as-History converts target camera trajectories into position-aligned, visibility-aware pseudo-history and feeds it through the model’s native visual-history pathway, optionally with one-video offline LoRA finetuning.

  • Results

    Across WorldScore, RE10K, and DAVIS, one-video-finetuned Warp-as-History is competitive with recent camera-control baselines trained on substantially more data, while improving camera adherence, visual quality, and consistency.

  • Takeaways & Limitations

    Warp-as-History exposes measurable zero-shot camera-following behavior in frozen history-conditioned models and enables low-resource finetuning that generalizes to unseen videos.

  • Takeaways & Limitations

    The method depends on externally reconstructed scene geometry, adding preprocessing and runtime costs while inheriting reconstruction errors in geometry, visibility, and disocclusion.

Abstract

from arXiv · show

Camera-controlled video generation has made substantial progress, enabling generated videos to follow prescribed viewpoint trajectories. However, existing methods usually learn camera-specific conditioning through camera encoders, control branches, or attention and positional-encoding modifications, which often require post-training on large-scale camera-annotated videos. Training-free alternatives avoid such post-training, but often shift the cost to test-time optimization or extra denoising-time guidance. We propose Warp-as-History, a simple interface that turns camera-induced warps into camera-warped pseudo-history with target-frame positional alignment and visible-token selection. Given a target camera trajectory, we construct camera-warped pseudo-history from past observations and feed it through the model's visual-history pathway. Crucially, we align its positional encoding with the target frames being denoised and remove warped-history tokens without valid source observations. Without any training, architectural modification, or test-time optimization, this interface reveals a non-trivial zero-shot capability of a frozen video generation model to follow camera trajectories. Moreover, lightweight offline LoRA finetuning on only one camera-annotated video further improves this capability and generalizes to unseen videos, improving camera adherence, visual quality, and motion dynamics without test-time optimization or target-video adaptation. Extensive experiments on diverse datasets confirm the effectiveness of our method.

1 Introduction

Warp-as-History exposes camera-following behavior in history-conditioned video models by presenting target-aligned, visibility-aware camera warps as visual pseudo-history. Zero-shot use reveals a weak capability in frozen models, while lightweight LoRA finetuning on one separate camera-annotated video stabilizes and generalizes it.

  • Motivation: Camera control must enforce prescribed trajectories while preserving appearance, disoccluding new content, and modeling foreground motion relative to the observer.These requirements make dynamic-video camera control more demanding than static novel-view synthesis.
  • Method: Warp-as-History converts target camera trajectories into target-frame-aligned, visibility-aware warped observations routed through the model’s native visual-history pathway.Source-visible regions provide history evidence, while newly revealed regions are left for the pretrained generator to complete.
  • Zero-shot capability: The imperfect warp cue elicits measurable zero-shot camera-following behavior from a frozen pretrained model, including scenes with substantial foreground motion.The effect is not robust enough as a final method, but reveals a latent camera-following capability.
  • Finetuning: Offline LoRA finetuning on one separate camera-annotated video stabilizes the exposed behavior and generalizes to unseen videos by amplifying an existing prior.The finetuning adjusts reliance on visible warp evidence, unreliable regions, and the generative prior for dynamics and disocclusion.
  • Evaluation: Experiments on WorldScore, RE10K, and DAVIS show that one-video-finetuned Warp-as-History is competitive with camera-control baselines trained on orders of magnitude more data.The reported comparison includes comparable camera adherence and strong visual-quality and consistency metrics.

2 Related Work

Prior camera-controlled video generation methods mainly use camera-matrix conditioning or warp- and geometry-based conditioning, while training-free approaches often incur inference-time control costs. Warp-as-History instead builds on native visual-history conditioning by packing camera-warped pseudo-history into the history stream and following the native sampler.

  • Camera-controlled video generation: Camera-controlled video generation has largely followed camera-matrix conditioning and warp- or geometry-conditioned approaches.Camera-matrix methods inject camera parameters through control branches, camera-aware attention, or positional encodings.
  • Training-free camera control: Training-free methods avoid camera-specific post-training but often use test-time optimization, denoising guidance, or latent repainting during inference.The passage identifies training-free camera control as an important comparison class while noting its inference-time costs.
  • Warp-as-History: Warp-as-History packs camera-warped pseudo-history into the native history stream, aligns it to target-frame positions, and applies visible-token selection.This interface conditions a video diffusion model on camera motion through its existing history pathway.
  • Warp-as-History: The method constructs camera-induced history once and follows the native sampler without per-sample optimization or extra denoising-time guidance.Its control interface therefore avoids additional optimization or guidance during sampling.
  • History-conditioned video generation: History-conditioned video generation uses previous frames as visual context, and recent methods study visual history and retrieved context for generation, rollout behavior, and scene consistency.Helios is identified as a recent state-of-the-art history-conditioned backbone with a native history interface.

3 Method

Warp-as-History represents a target camera trajectory as camera-warped pseudo-history routed through the pretrained model’s native visual-history pathway. It aligns warp tokens with target denoising frames, retains only visible evidence, and optionally improves the frozen-model behavior through lightweight one-video LoRA finetuning.

  • Conditioning interface: Warp-as-History converts a target camera trajectory into camera-warped pseudo-history and feeds it through the backbone’s native history pathway.The same conditioning interface supports both the frozen-model diagnostic and the one-video finetuned model.
  • Camera-warped pseudo-history: The method reconstructs the scene, projects it to each target camera, and uses the resulting 2D warp video as visual history evidence rather than a hard render target.The warp is obtained by projecting an off-the-shelf reconstruction model’s scene reconstruction along the target trajectory.
  • Target-frame positional alignment: Target-frame positional alignment makes each warp frame serve as evidence for the corresponding frame being denoised instead of ordinary past context.The alignment addresses the pretrained model’s separation between history and the current noisy chunk through temporal positional embeddings.
  • Visible-token selection: Visible-token selection drops invalid warp tokens caused by disocclusions, holes, stretched textures, or unreliable geometry, leaving missing regions for generation.The selection is applied after native history construction, while the backbone’s history operator remains unchanged.
  • One-video finetuning: Lightweight LoRA finetuning on one separate camera-annotated video adjusts the history reader’s balance between visible warp cues and the generative prior.The unchanged video-generation loss optimizes only the low-rank update to improve independent dynamics and disocclusion completion.

4 Analysis and Experiments

Experiments evaluate Warp-as-History across static, real-scene, and dynamic-video benchmarks, testing camera controllability, visual quality, interface choices, and finetuning-data effects. The method improves camera control and visual quality without test-video adaptation, including under cross-domain evaluation.

  • Benchmark evaluation: Experiments span WorldScore, RE10K, and DAVIS, covering static generation, real static scenes with camera motion, and dynamic foreground motion.Ours (one-shot) denotes one LoRA run on the DAVIS car-roundabout video without per-test-video adaptation.
  • WorldScore comparison: 134.7% relative camera-control gain is achieved after one-shot finetuning on WorldScore, rising from 26.42 for text-only Helios-Distilled to 62.00.The zero-shot setting reaches 61.32, corresponding to a 132.1% relative gain.
  • RE10K and DAVIS comparisons: On RE10K, Ours remains in a comparable camera-following range despite training on DAVIS, while achieving the best DOVER, Subject Consistency, Background Consistency, and Imaging scores.External baselines use large-scale training data from the real-estate domain, creating a domain mismatch for the one-shot update.
  • RE10K and DAVIS comparisons: On DAVIS, Ours achieves the best FID/FVD, Subject Consistency, and Background Consistency while retaining camera-following accuracy comparable to external baselines.Qualitative examples also show cleaner preservation of scene content and foreground motion than prior warp-based baselines, which often exhibit artifacts, blur, or distorted objects.
  • Ablations and data scaling: Interface ablations compare NoAlign, NoVisDrop, and the full interface across zero-shot and one-shot regimes, while source-selection and few-shot analyses treat data scaling as diagnostics.The clearest gain occurs from zero-shot invocation to one-video finetuning; adding more videos produces smaller, non-monotonic changes.

5 Conclusion

Warp-as-History routes camera-warped pseudo-history through a frozen model’s native history pathway, using target-frame positional alignment and visible-token selection. It enables zero-shot camera following and improves with lightweight one-video LoRA finetuning without test-time optimization or per-video fitting.

  • Method: Warp-as-History routes camera-warped pseudo-history through the model’s native history pathway with target-frame positional alignment and visible-token selection.This interface is the method’s core design.
  • Zero-shot capability: The interface reveals zero-shot camera-following behavior in a frozen model.No model training is required for this capability.
  • Finetuning and evaluation: A lightweight LoRA update on one separate video stabilizes camera following without test-time optimization or per-video fitting.The method was evaluated across WorldScore, RE10K, and DAVIS.

A HyWorldPlay Evaluation Details

Table 2 evaluates HyWorldPlay on 50 randomly sampled WorldScore images using three random camera directions per image, 30-second generations, and VBench, separately from Table 1’s native 33-frame evaluation.

  • Evaluation protocol: Table 2 samples 50 WorldScore images, generates 30-second videos for three random camera directions per image, and evaluates them with VBench.This protocol is separate from the native 33-frame WorldScore evaluation used in Table 1.

B Additional Qualitative Comparisons

This section presents additional qualitative comparisons with external camera-control methods on in-the-wild videos. The comparison uses the same column layout as Figure 4.

  • Additional Qualitative Comparisons: Figure 7 compares the method qualitatively with external camera-control methods.The comparison is presented in Figure 7.
  • Additional Qualitative Comparisons: The qualitative comparison is conducted on in-the-wild videos.These videos provide the evaluation setting for Figure 7.
  • Additional Qualitative Comparisons: Figure 7 uses the same column layout as Figure 4.The caption explicitly states that the columns follow Figure 4's layout.

C Interface Ablation Settings and Full Tables

This section specifies the evaluation protocols, implementation details, row definitions, and auxiliary sensitivity analyses supporting the interface-ablation and external-baseline results. It emphasizes matched evaluation conditions and reports multi-video finetuning only as a non-monotonic sensitivity check.

  • Evaluation protocol: All methods in each table use the same evaluated videos and target camera trajectories, with training sequences disjoint from evaluation data.Any compute-limited evaluation subset is randomly selected once before evaluation and then fixed across compared rows.
  • Implementation details: LoRA is trained on Helios-Mid and mounted on the distilled checkpoint, whose six denoising steps are grouped 2 + 2 + 2 across three resolution stages.Aligned warp history and LoRA apply only in the first stage; later higher-resolution stages remain unchanged.
  • Interface ablations: Interface-ablation rows use Helios-Distilled; Full combines warp-as-history with RoPE alignment and visible-token dropping, while NoAlign, NoVisDrop, SeqConcat, and ChFusion isolate alternatives.The text-only baseline receives no camera-control condition, but camera metrics use the same target reference as camera-conditioned rows.
  • Dataset protocols: WorldScore evaluates 32 deterministic static_cc_dev32 samples under native 33-frame evaluation, spanning 2 visual styles, 2 scene types, and 8 single-camera motions.Every listed row is evaluated on the same 32 samples.
  • Dataset protocols: DAVIS uses 77 videos with a common 33-frame first-chunk protocol, while RE10K uses a fixed 100-sequence DAVIS-aligned ablation subset and omits unavailable source-report metrics.The DAVIS camera condition uses Pi3X-estimated pseudo-ground-truth trajectories from the original videos.
  • Additional-data sensitivity: Increasing the finetuning set up to 12 videos does not yield a clear monotonic improvement over one-video finetuning, so multi-video runs serve only as a sensitivity check.Table 6 reports DAVIS+RE10K mean metrics, while Tables 7 and 8 provide full per-dataset report cards without per-test-video adaptation on RE10K.

D Supplementary One-Shot Source Diagnostics

The supplementary section diagnoses one-shot source sensitivity by sweeping source videos under a fixed LoRA recipe and evaluating reconstruction, camera-following, distribution, and quality metrics across DAVIS and RE10K. It also profiles camera, visibility, motion, and additional interface-ablation characteristics.

  • Evaluation axes: The sweep records reconstruction, perceptual, camera-following, distribution, and quality metrics, including PSNR, SSIM, LPIPS, VisLPIPS, R-Err, T-Err, FID/FVD, DOVER, and VBench axes.VisLPIPS denotes visible-region LPIPS; Dyn. and Img. denote VBench Dynamic Degree and Imaging Quality.
  • Source diagnostics: For each source clip, the diagnostics profile invisible-region ratios, camera rotation, translation-direction angle, foreground centroid motion and area, and source/target characteristics.The supplied passage lists mean and maximum values for invisible ratio and camera rotation.
  • One-shot source sensitivity: The one-shot source sweep changes only the source video while keeping the LoRA recipe fixed, and aggregates a lower-is-better mean rank across DAVIS and RE10K.Table 9 presents a compact subset of the sweep, while Tables 10 and 11 report the full dataset-specific results.
  • DAVIS source sweep: Table 10 evaluates source-video LoRA updates on DAVIS, with rows differing only in the source video used for one-shot finetuning.The table includes reconstruction, camera-following, distribution, and quality axes.
  • RE10K source sweep: Table 11 evaluates the same source-video LoRA updates on RE10K without per-test-video adaptation.Rows use the same source-video updates as Table 10.

E Runtime Analysis

Warp-as-History increases runtime because camera-warped pseudo-history adds transformer tokens, with overhead governed mainly by the fraction of visible warped-history tokens. On one NVIDIA A800 GPU, reducing visibility from 86% to 47% lowers both transformer/sampling and end-to-end overhead.

  • Runtime overhead: Warp-as-History increases transformer token processing by adding camera-warped pseudo-history to the native history stream.Runtime is measured for generating one 33-frame chunk on a single NVIDIA A800 GPU.
  • Runtime overhead: With 86% visible tokens, transformer/sampling overhead increases by 7.59s and end-to-end runtime increases by 7.81s.The transformer/sampling stage accounts for almost all of the end-to-end increase.
  • Visibility filtering: With 47% visible tokens, transformer/sampling overhead is 3.38s and end-to-end overhead is 4.62s.Invisible-token dropping reduces the number of warped-history tokens retained after visibility filtering.
  • Runtime overhead: Camera rendering, warp VAE encoding, and warp/mask preparation together contribute only about 1–2 seconds of overhead.The main runtime cost comes from transformer/sampling rather than geometry or warp preparation.

F Limitations · G Broader Impacts

Warp-as-History’s main limitation is that its warp-construction step adds preprocessing cost and depends on external reconstruction quality. The method may support creative and scientific applications, but deployment must address potential misuse and consent, labeling, and safety concerns.

  • F Limitations: Warp construction adds preprocessing cost because it uses an external reconstruction model online.The reconstruction is projected to target future cameras.
  • F Limitations: The method inherits reconstruction failures in geometry, visibility, and disoccluded regions.These failures arise from reconstructing the observed scene before projecting it to target cameras.
  • F Limitations: Warp-as-History avoids training a camera-specific control branch but does not eliminate preprocessing dependence.Its interface shifts part of the system requirement to online warp construction.
  • G Broader Impacts: The work may benefit creative editing, virtual cinematography, simulation, and controllable scene visualization.These applications are identified for camera control in pretrained video generation models.
  • G Broader Impacts: The technology could be misused to create misleading videos or alter private or sensitive footage without consent.These risks are presented as broader concerns shared with video generation and editing methods.
  • G Broader Impacts: Deployment should follow the underlying video model’s safety policies and clearly label generated or edited content.These practices are recommended for any release or deployment.
Loading 2605.15182v1…