PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Joint Spatial-Temporal Transformations for Video Inpainting

Yanhong Zeng, Jianlong Fu, Hongyang Chao

arXiv:2007.10247v1cs.CV

TL;DR

High-quality video inpainting is challenged by inconsistent attention and temporal artifacts when existing methods process frames individually. This paper introduces STTN, which jointly completes all input frames with spatial-temporal self-attention and adversarial optimization, reporting state-of-the-art results on challenging masks, including 2.4% PSNR and 19.7% VFID relative improvements.

  • Problem

    Existing attention-based video inpainting methods process frames frame by frame, risking inconsistent spatial-temporal attention and artifacts under complex motion.

  • Method

    STTN formulates inpainting as a multi-to-multi task, using multi-scale patch-based spatial-temporal self-attention and a spatial-temporal adversarial loss.

  • Results

    2.4% PSNR and 19.7% VFID relative improvements over state-of-the-art models are reported on video inpainting evaluations.

  • Takeaways & Limitations

    STTN simultaneously completes all input frames while learning perceptually pleasing and spatially-temporally coherent contents.

  • Takeaways & Limitations

    STTN may generate blurs for large missing masks during continuous quick motions because short-term complex-motion continuity is hard to capture without 3D representations.

Abstract

from arXiv · show

High-quality video inpainting that completes missing regions in video frames is a promising yet challenging task. State-of-the-art approaches adopt attention models to complete a frame by searching missing contents from reference frames, and further complete whole videos frame by frame. However, these approaches can suffer from inconsistent attention results along spatial and temporal dimensions, which often leads to blurriness and temporal artifacts in videos. In this paper, we propose to learn a joint Spatial-Temporal Transformer Network (STTN) for video inpainting. Specifically, we simultaneously fill missing regions in all input frames by self-attention, and propose to optimize STTN by a spatial-temporal adversarial loss. To show the superiority of the proposed model, we conduct both quantitative and qualitative evaluations by using standard stationary masks and more realistic moving object masks. Demo videos are available at https://github.com/researchmm/STTN.

1 Introduction

Video inpainting fills missing regions with plausible contents, but existing methods face challenges from limited temporal receptive fields and inconsistent attention under complex motion. STTN addresses these issues by jointly transforming spatial and temporal information, with adversarial optimization and evaluations on stationary and moving masks.

  • Motivation: Video inpainting fills missing regions with plausible contents for restoration, object removal, retargeting, and exposure restoration applications.High-quality results remain difficult because of limited high-level video understanding and high computational complexity.
  • Existing limitations: 3D convolutional and recurrent approaches aggregate nearby frames but can produce temporal artifacts because of limited temporal receptive fields.Attention methods instead exploit long-range correspondences from distant frames, but complex motion causes major limitations.
  • Proposed method: STTN formulates inpainting as a multi-to-multi task that simultaneously fills missing regions in all input frames using multi-scale patch-based spatial-temporal attention.Different transformer heads match spatial patches across scales, while stacked layers refine attention using updated region features.
  • Optimization: A spatial-temporal adversarial loss jointly optimizes STTN for perceptually pleasing and coherent video contents.The loss complements joint spatial-temporal transformation learning in the deep generative model.
  • Evaluation: 2.4% PSNR and 19.7% VFID relative improvements over state-of-the-art models are reported, alongside evaluations using stationary and moving object masks.The paper also reports qualitative and quantitative evaluations and extensive ablation studies.

2 Related Work

Video inpainting research spans patch-based optimization and deep generative models, with later attention methods targeting long-range correspondences. However, frame-wise attention and recursive processing can struggle with complex motion and temporal consistency, motivating joint spatial-temporal modeling.

  • Patch-based methods: Early patch-based methods synthesize missing contents by sampling similar spatial or spatial-temporal patches from known regions through global optimization.Some methods additionally use foreground and background segmentation or jointly estimate appearance and optical flow.
  • Deep generative models: Deep generative models combine 3D and 2D convolutions or recurrent networks to model temporal information and spatial details, but complex scenes can remain blurry.Later work improves performance by jointly estimating appearance and optical flow.
  • Attention methods: Attention-based methods model long-range correspondences by weighted summation of aligned frames or pixel-wise recursive hole filling.These approaches use reference contents from other frames to synthesize missing regions.
  • STTN: STTN instead uses multi-layer, multi-head spatial-temporal transformers to fill holes simultaneously across all input frames.The architecture contains a frame-level encoder, stacked transformers, and a frame-level decoder.
  • Attention methods: Frame-wise attention can be difficult to apply under complex motion, while recursive attention may not ensure consistent results across iterations.Existing attention models also process videos frame by frame without specially designed temporal-coherence optimization.

3 Spatial-Temporal Transformer Networks

STTN formulates video inpainting as a multi-to-multi task that jointly completes all input frames using neighboring and distant context. Its spatial-temporal transformers retrieve coherent contents across frames and spatial scales, while reconstruction and adversarial objectives target accuracy, perceptual quality, and coherence.

  • Overall design: The multi-to-multi formulation jointly completes all input frames from neighboring and uniformly sampled distant frames.Distant frames can cover key moments and provide whole-video context, while neighboring frames support temporal consistency.
  • Spatial-temporal transformer: Multi-head patch-based attention searches coherent contents across spatial and temporal dimensions using patches at different scales.Different patch sizes help address appearance changes caused by complex motion, while matching patches across frames supports retrieval of relevant regions.
  • Spatial-temporal transformer: The transformer embeds frame features into query, key, and value spaces, matches patches across all frames, and reconstructs outputs by weighted summation of relevant value patches.A residual block further enhances the assembled features using within-frame context.
  • Spatial-temporal transformer: Stacked transformer layers iteratively improve missing-region features in one feed-forward process, learning coherent spatial-temporal transformations.The paper illustrates this behavior by showing attention maps that track a moving dog across the video.
  • Optimization objectives: STTN is optimized with hole and valid-region L1 reconstruction losses together with a Temporal PatchGAN adversarial loss.The objectives are designed to support per-pixel accuracy, perceptual rationality, and spatial-temporal coherence.

4 Experiments

Experiments evaluate STTN on YouTube-VOS and DAVIS using stationary and moving free-form masks, with quantitative, qualitative, user-study, attention, and ablation analyses. STTN achieves stronger reported reconstruction quality and visual results, while multi-scale patches and eight transformer layers improve the model and large masks with fast motion remain challenging.

  • Evaluation setup: Experiments use YouTube-VOS and DAVIS with stationary and moving free-form masks to simulate applications including watermark and object removal.The evaluation compares recent competitive approaches and reports PSNR, SSIM, flow warping error, and VFID.
  • Quantitative evaluation: STTN improves over state-of-the-art models on stationary-mask reconstruction, especially in PSNR, flow warp error, and VFID.On YouTube-VOS, the reported relative improvements are 2.4%, 1.3%, and 19.7%, respectively.
  • Qualitative evaluation: Visual comparisons report more coherent structures and details than competing methods for stationary masks, and visually pleasing results against CAP under complex moving-mask scenes.The moving-mask examples include clearer faces and a better result than CAP in one frame.
  • Attention analysis: STTN attention maps select relevant objects for partially occluded-object completion and relevant backgrounds for background completion.The visualized regions are the top three regions selected according to attention weights.
  • User study: In a user study, the model is ranked first in most cases.The study uses 28 volunteers and evaluates stationary-mask and moving-mask results from DAVIS and YouTube-VOS.
  • Ablation study: Multi-scale patch representations outperform single-head alternatives, while stacking transformers yields continuous improvements with the best results at eight layers.The full model combines four patch scales, and eight layers are used in the final model.
  • Failure case: For a large mask covering a dancing woman, STTN fails to generate continuous motion and produces blur inside the mask.The authors associate this failure with difficulty capturing short-term continuity of complex motion using spatial patches alone.

5 Conclusions

The paper concludes that joint spatial-temporal transformation learning, multi-scale patch representations, and spatial-temporal adversarial loss enable efficient video completion with state-of-the-art results. It also identifies blurring during large-mask reconstruction of continuous quick motions as a limitation and proposes 3D spatial-temporal attention as future work.

  • Conclusion: Joint spatial-temporal transformation learning with multi-scale patch representations and spatial-temporal adversarial loss supports efficient completion of all input frames.The model is evaluated on challenging free-form masks from YouTube-VOS and DAVIS.
  • Limitation: STTN may generate blur and discontinuous motion when large missing masks contain continuous quick motions.The authors plan to use attention on 3D spatial-temporal patches and investigate other temporal losses.

A Video Inpainting Results

The paper evaluates video inpainting visually across stationary and moving masks using comprehensive video comparisons. STTN outperforms the selected state-of-the-art models in most cases.

  • Sampled static frames may look less blurry while concealing stronger artifacts in dynamic videos.The authors therefore provide 20 video cases for more comprehensive comparisons.
  • The evaluation covers 90 DAVIS videos and 508 Youtube-VOS videos, with 20 cases randomly shown for visual comparison.Five DAVIS and five Youtube-VOS cases test stationary masks, while ten DAVIS videos test moving masks.
  • STTN outperforms the competitive LGTSM and CAP models in most visual comparison cases.The baselines were fine-tuned multiple times using their publicly provided codes and models.

B Stationary Mask Generation Algorithm

The stationary mask generator creates irregular closed contours from randomly sampled control points. Cubic Bezier curves connect the points smoothly and close the contour.

  • The algorithm randomly generates control points around a unit circle and connects them into a closed cyclic contour.The contour is intended to simulate stationary masks such as watermarks.
  • The mask begins as a zero-valued image, with point count and starting coordinates sampled within image and hyper-parameter ranges.maxPointNum and maxLength control the number and extent of generated mask geometry.
  • For each sampled angle and length, the algorithm computes coordinates and connects successive points using cubic Bezier curves.A final Bezier connection joins the last point to the original point, ensuring a closed contour.

C Details of Network Architecture

STTN uses a generative adversarial architecture with STTN as generator and T-PatchGAN as discriminator. The network and discriminator are specified through convolutional and stacked transformer layers.

  • STTN serves as the generator in a generative adversarial framework, while T-PatchGAN acts as its discriminator.T-PatchGAN classifies spatial-temporal features as real or fake, and STTN learns to fool it.
  • The STTN architecture stacks eight proposed spatial-temporal transformer layers alongside 2D convolution layers and bilinear upsampling.Each transformer layer uses 1 × 1 and 3 × 3 convolutions.
  • T-PatchGAN consists of six 3D convolution layers, with spectral normalization used to stabilize GAN training.The discriminator classifies each spatial-temporal feature as real or fake.

D Implementation details

Implementation uses fixed-resolution video frames, specified optimization settings, and a multi-scale patch representation designed for efficient training and inference. The reported model has measurable memory, speed, and parameter costs.

  • All video frames are resized to 432×240, with batch size 8 and learning rate 1e-4 decayed by factor 0.1 every 150k iterations.Training samples five frames per iteration, consecutively or discontinuously with equal probability.
  • The full model has 12.6M trainable parameters and uses about 3.9G GPU memory per DAVIS video completion.The reported average speed is 24.3fps on an NVIDIA V100 and 10.43 fps on an NVIDIA P100.
  • Equation (11) focuses on spatial-temporal transformer computation complexity using D, n, HW, pwph, k_l, and C.The analysis omits other costs, including encoding and decoding, for simplification.

E More ablation studies

Ablations show that distant-frame context, masked normalization, and T-PatchGAN loss each improve STTN video inpainting, particularly for temporal quality and large missing regions.

  • Effectiveness of utilizing distant frames: 5.70% relative VFID improvement comes from leveraging visible contexts in distant frames.The full model uses a sampling rate of 10 to avoid redundant frames and reduce computation costs.
  • Effectiveness of masked normalization: Masked normalization substantially improves similarity calculation compared with omitting the operation.The operation normalizes similarity by vector dimension and excludes unknown regions from similarity computation.
  • Effectiveness of the Temporal PatchGAN Loss: T-PatchGAN jointly promotes local-global perceptual rationality and spatial-temporal coherence, including when attention models use discontinuous frames.The authors introduce T-PatchGAN into attention-based video inpainting models and report that it remains effective beyond consecutive-frame inputs.
  • Effectiveness of the Temporal PatchGAN Loss: 6.9% relative VFID improvement is achieved by replacing style and perceptual losses with T-PatchGAN loss.T-PatchGAN also produces more coherent results, supporting joint spatial-temporal adversarial learning in STTN.
  • Effectiveness of the Temporal PatchGAN Loss: Perceptual and style losses perform similarly to T-PatchGAN in small missing regions but generate artifacts in large missing regions.Because these losses use single-frame features, they cannot leverage temporal contexts; T-PatchGAN produces fewer artifacts in the cited comparison.
Loading 2007.10247v1…