Source-linked AI summary
HL-OutPaint: Coarse-to-Fine Video Outpainting for High-Resolution Long-Range Videos
Jeongeun Park, Janghyeok Han, Geonung Kim, Hyun-Seung Lee, Kyuha Choi, Youngseok Han, Sunghyun Cho
TL;DR
Large spatial extrapolation over long videos remains difficult because existing approaches often address only one dimension or lose fine-grained temporal cues during compression. HL-OutPaint uses coarse-to-fine Global Coarse Guidance with global-local frame swapping, followed by guided high-resolution outpainting, and reports stable performance across challenging wide-expansion, long-sequence scenarios.
Problem
Video outpainting needs to support large spatial extrapolation over long sequences while preserving global spatio-temporal coherence, but existing methods address only part of this challenge and compressed guidance loses fine temporal cues.
Method
HL-OutPaint constructs globally coherent low-resolution GCG using global-local frame swapping, then performs GCG-guided high-resolution outpainting with tile-based diffusion.
Results
HL-OutPaint achieves state-of-the-art performance across scenarios requiring wide spatial expansion and long-range temporal consistency.
Takeaways & Limitations
The framework provides stable, coherent outpainting for large spatial expansions over extended video sequences by combining global structure with local temporal information.
Takeaways & Limitations
HL-OutPaint is not suitable for real-time applications and may fail in extremely large or long cases because aggressive downsampling can lose high-frequency details in outpainted regions.
Abstract
from arXiv · showhide
Video outpainting generates plausible visual content beyond the original spatial extent of a video, playing a key role in adapting videos to diverse display formats. To support such use cases, it must enable large spatial extrapolation over long sequences. However, most existing methods address only one of these challenges or lack explicit mechanisms for ensuring global spatio-temporal consistency, leading to notable limitations. In this paper, we propose HL-OutPaint, a high-resolution video outpainting framework for long sequences. Our approach follows a coarse-to-fine strategy with a two-stage pipeline. We first construct Global Coarse Guidance (GCG), a low-resolution representation that captures global structure and dominant motion across the video. Unlike naive downsampling, GCG is built via a novel global-local frame swapping mechanism that couples sparse global keyframes with local temporal windows and exchanges information during sampling. This enables GCG to encode both long-term structural consistency and short-term temporal dynamics in a unified representation. Guided by this representation, HL-OutPaint then performs high-resolution outpainting to generate spatially detailed and temporally consistent content. By separating global structure modeling from fine-grained synthesis, our framework achieves stable, coherent generation for large spatial expansion and long video sequences. Extensive experiments show that HL-OutPaint outperforms existing methods in challenging scenarios involving wide spatial extrapolation and long video sequences.
1 Introduction
HL-OutPaint targets video outpainting over large spatial extents and long sequences, where existing methods typically address only one dimension or lose global spatio-temporal coherence. Its coarse-to-fine design combines globally coherent guidance with local temporal detail for high-resolution synthesis.
- Video outpainting supports adapting fixed-aspect-ratio content to diverse displays and editing tasks such as reframing, stabilization, and overlay placement.
- Existing generative approaches demonstrate quality at predefined resolutions and limited sequence lengths, leaving a gap for larger spatial extrapolation over longer videos.
- Infinite-Canvas handles large spatial expansion but can lose global coherence, whereas M3DDM targets long sequences yet struggles with rapid motion between sparse keyframes.
- HL-OutPaint constructs a spatio-temporally low-resolution Global Coarse Guidance representation before high-resolution tile-based refinement.The guidance models the full sequence holistically within the diffusion model's attention span, establishing a consistent structural foundation.
- Global-local frame swapping exchanges information between sparse global keyframes and local temporal windows, preserving global stability while recovering fine-grained temporal observations.The mechanism helps keyframes inherit details such as objects appearing or disappearing within local windows.
- Experiments report state-of-the-art performance across scenarios requiring wide spatial expansion and long-range temporal consistency.
2 Related Work
Related work spans image and video outpainting, inpainting, and autoregressive video generation, but prior methods generally cover either spatial expansion or long temporal horizons rather than both.
- Image outpainting methods provide spatial extrapolation, but directly extending them to video can cause flickering because they lack temporal modeling.
- M3DDM uses keyframe-based generation for long-sequence outpainting, while Infinite-Canvas supports large spatial expansion through global positional guidance.
- M3DDM+, OutDreamer, and related methods extend temporal duration but do not support substantial spatial extrapolation.
- Video inpainting generates content in arbitrary masked regions and can be viewed as a generalized form of outpainting, although outpainting poses greater challenges.
- Autoregressive video generation extends videos by conditioning on previous outputs, but sequential prediction can suffer from error accumulation.
3 Preliminary: Video Outpainting and Diffusion Prior 𝑓}𝐹
Video outpainting extends an input video into masked spatial regions using a video outpainting function and diffusion-based generation. Fixed-resolution, short-window diffusion models motivate HL-OutPaint's compressed guidance and tiled processing strategy.
- An original video is padded from H′ × W′ to H × W, and the goal is to synthesize content in the appended regions.
- The binary mask M marks outpainting regions with 1 and observed regions with 0 in each frame.
- Video diffusion models synthesize coherent spatio-temporal content conditioned on partially observed video regions and masks.
- The diffusion operator D performs latent-space denoising and is used for keyframe denoising, local-window refinement, and multi-scale guidance construction.
- Existing video diffusion models operate at fixed spatial resolutions and short temporal windows, while spatio-temporal tiling can introduce cross-tile inconsistencies without global context.
- HL-OutPaint addresses these constraints with a two-stage framework that constructs GCG and then performs GCG-guided outpainting using tile-based diffusion.
4 HL-OutPaint
HL-OutPaint uses a coarse-to-fine pipeline that first builds globally coherent guidance and then performs guided high-resolution outpainting. Its global-local frame swapping and overlapping spatio-temporal processing preserve long-range structure alongside local temporal detail.
- GCG Construction: HL-OutPaint constructs Global Coarse Guidance from sparse keyframes and local temporal windows to capture global structure and short-range dynamics.Keyframes emphasize global structure, while local windows preserve fine-scale temporal cues during parallel denoising.
- GCG Construction: Global-local frame swapping exchanges corresponding latent frames during denoising, coupling global context with fine-scale temporal information.The strategy replaces keyframe latents with matching local-window latents during early denoising steps, allowing information to propagate through later steps.
- GCG Construction: For very long videos, multi-scale guidance construction inserts temporal midpoint keyframes to provide denser coverage.The initial uniformly sampled keyframes are refined by adding keyframes between existing ones.
- Model Adaptation: Separate diffusion models are finetuned for sparse-frame GCG construction and densely sampled GCG-guided outpainting.The two stages operate under different temporal regimes and therefore require distinct adaptations.
- GCG-guided Video Outpainting: GCG-guided outpainting first completes missing regions temporally at reduced resolution, then restores spatial detail through high-resolution refinement.The low-resolution guidance supplies corresponding frames during temporal completion before the result is upsampled and regenerated with diffusion.
- GCG-guided Video Outpainting: Overlapping temporal and spatio-temporal tiles enable diffusion processing beyond the model’s sequence and resolution capacity while blending boundaries for consistency.Temporal tiles are blended after each diffusion step, and spatial refinement uses overlapping tiles to maintain consistency across adjacent regions.
5 Experiments
HL-OutPaint is evaluated across datasets, spatial expansion scales, and video lengths using quantitative and qualitative comparisons. Ablations show that global-local frame swapping and joint spatial-temporal GCG compression improve coherence, while spatial refinement restores high-resolution detail.
- Experimental Setup: Experiments span DAVIS, DAVIS-20, YouTube-VOS, Long-Video, and Short-Form datasets, with comparisons reported across spatial extrapolation settings and a user study.DAVIS includes videos ranging from 25 to 104 frames, with evaluation from 512 × 512 to 1280 × 720.
- Baseline Comparisons: Qualitative comparisons show severe artifacts from MOTIA and M3DDM under large spatial extrapolation, while Infinite-Canvas and VACE often lose long-term temporal coherence.The comparisons cover DAVIS, Short-Form, and Long-Video datasets.
- Baseline Comparisons: HL-OutPaint achieves the best performance across most evaluation metrics in Table 1 and preserves long-term consistency in challenging occlusion scenarios.On a train-platform example, competing methods produce inconsistent appearances before and after occlusion, whereas HL-OutPaint maintains consistency.
- Ablation & Analysis: Global-local frame swapping injects nearby-frame structural cues into sparsely sampled keyframes, resolving hallucinated inconsistencies and improving all metrics on Long-Video.A partially cropped traffic sign is recovered from an arrow-shaped observation in neighboring frames rather than an arbitrary hallucination.
- Ablation & Analysis: Joint spatial and temporal compression avoids the long-term incoherence caused by independent temporal tiles and the repetition artifacts caused by independently generated spatial tiles.Without temporal compression, objects such as goal posts can disappear; without spatial compression, adjacent regions can repeat.
- Ablation & Analysis: Spatial refinement restores text shapes and complex textures more clearly than bicubic upsampling, improving realism in grass and cow-fur regions.The refinement stage transforms temporally completed low-resolution results into high-resolution videos.
6 Conclusion
HL-OutPaint uses a coarse-to-fine framework to preserve spatio-temporal coherence during large spatial extrapolation over long sequences, but extreme cases remain constrained by downsampling and coverage limits.
- HL-OutPaint constructs Global Coarse Guidance and refines it into spatially detailed, temporally consistent high-resolution results.
- The global-local frame swapping mechanism supports both global and local temporal coherence.
- The method maintains stability across diverse challenging scenarios involving large spatial expansion and extended video sequences.
- Extreme spatial expansion can lose high-frequency details because GCG construction requires heavy input downsampling, producing smooth or blurry outpainted regions.
- For very long videos, insufficient keyframe and local-window coverage can make temporal consistency difficult to maintain.
Supplementary Material
The supplementary material covers tiled denoising and multi-scale GCG construction, including iterative refinement for videos with large temporal gaps.
- The supplementary material includes details on spatio-temporal tiling, multi-scale GCG construction, implementation, hyperparameter analysis, and compression ablations.
- Overlapping spatio-temporal tiles are independently denoised and blended to reduce boundary artifacts.
- Center-weighted blending assigns higher weights near tile centers and lower weights near boundaries, enabling smooth transitions.
- For significantly long videos, GCG is iteratively refined until the maximum temporal distance between adjacent keyframes falls below a predefined threshold.
B.2 Training for Multi-scale GCG.
Multi-scale GCG training simulates sparse keyframe conditions so the model can construct coherent guidance across different temporal scales.
- Training randomly samples keyframes from densely sampled videos and removes intermediate frames to create large temporal gaps.
- Varying the sampling interval exposes the model to multiple temporal sparsity levels and improves robustness during GCG construction.
- The model is trained on approximately 17,000 OpenVid-1M videos resampled to 768×768 resolution and 49 frames.
C.2 Stage-wise LoRA Training
Stage-wise training separates spatial-only GCG construction from spatio-temporal high-resolution outpainting, while experiments examine frame swapping, stride, interval, and compression choices.
- The two inference stages share a frozen video diffusion transformer but use stage-specific LoRA modules.
- Spatial-only compression independently encodes sparse keyframes, preserving representations suited to temporally discontinuous frame-wise background generation.
- The GCG-guided stage uses temporally dense sequences with standard spatio-temporal compression to model motion and scene dynamics jointly.
- Global-local frame swapping is most effective early in denoising because it propagates local temporal cues while global structure is established.
- Applying swapping during the first 8 of 40 denoising steps achieves the best overall performance, while excessive application can disrupt fine-detail refinement.
- Stride can be selected from motion, with smaller values for faster motion, while an interval of 20 balances temporal coverage and efficiency.
- Removing global spatial compression often produces repetition artifacts and structural inconsistency across adjacent regions.
E.3 Quantitative Analysis
The ablation shows that full spatio-temporal compression provides the strongest overall balance across visual fidelity and temporal consistency metrics. The section also describes hierarchical training and inference procedures that adapt a pretrained diffusion model for long-video outpainting.
- Quantitative ablation: Full spatio-temporal compression achieves the strongest overall performance across most metrics, while spatial-only compression favors PSNR and SSIM and temporal-only compression favors SC.This ablation evaluates the separate and combined contributions of spatial and temporal compression in Global Coarse Guidance.
- Optimization: The model adapts a pretrained video diffusion prior by training inserted LoRA parameters while keeping most original model parameters frozen.Stage-wise sample preparation conditions denoising on masked video latents, masks, prompts, timesteps, and anchor frames.
- Training procedure: HL-OutPaint trains in two stages, first on 13 sparsely sampled keyframes for spatial completion and then on full clips for longer-sequence motion coherence.The second stage uses longer-stride anchor frames and temporally compressed video-VAE processing.
- Inference procedure: The inference pipeline constructs sparse-to-dense guidance before applying full-resolution outpainting, preserving long-range temporal structure while maintaining local spatial detail.Sparse guidance is progressively refined with synthesized midpoint frames before dense generation.
H Analysis of SC and BC Metrics
The evaluation examines spatially localized temporal-consistency metrics, inference efficiency, and perceptual quality through a user study. HL-OutPaint is reported as the fastest compared method in the tested long-video setting.
- Metric analysis: SC and BC are computed over spatial tiles to capture localized consistency differences that one-dimensional global features may miss.The spatially aware evaluation divides each frame into tiles before computing the consistency metrics independently.
- Inference efficiency: 105 minutes versus 143 minutes for VACE: HL-OutPaint has the fastest inference time on a 500-frame 720 × 1280 video using an A100-80GB GPU.VACE is identified as the second-fastest baseline in this comparison.
- User study: The user study compares HL-OutPaint with four baselines across visual quality, temporal consistency, subject quality, and background quality.The study includes 20 participants and 10 randomly selected videos, reporting vote percentages.
- Design assumption: Temporal RoPE receives no explicit modification because fine-tuning adapts the diffusion backbone to sparse keyframe sequences using the original positional encoding.This is the stated handling of temporal positional encoding in the first stage.
L Dataset Details
The experiments use publicly accessible Pexels videos organized into separate long-video and short-form datasets. Corresponding video lists are provided for both collections.
- Data source: All videos are collected from Pexels under its free license, with corresponding URLs provided.The data are publicly accessible and expected to remain available.
- Dataset organization: The experimental data are divided into a Long-Video dataset and a Short-Form dataset.The two dataset listings are provided in Tables 7 and 8, respectively.