PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust High-Resolution Video Matting with Temporal Guidance

Shanchuan Lin, Linjie Yang, Imran Saleemi, Soumyadip Sengupta

arXiv:2108.11515v1cs.CV

TL;DR

Human video matting needs robust, real-time background replacement without trimaps or pre-captured backgrounds. The paper uses recurrent temporal guidance and joint matting-segmentation training, achieving state-of-the-art performance with 58% of the parameters and 4K 76 FPS and HD 104 FPS on a GTX 1080Ti.

  • Problem

    Existing neural matting methods can be non-robust, artifact-prone, and frame-independent, while auxiliary inputs are undesirable for broad human video applications.

  • Method

    The method uses a recurrent architecture for temporal information and jointly trains matting and semantic segmentation objectives without auxiliary inputs.

  • Results

    58% parameters, 4K 76 FPS, and HD 104 FPS are reported while outperforming the previous state-of-the-art method.

  • Takeaways & Limitations

    Temporal information and joint segmentation training are reported to improve matting quality, consistency, and robustness across videos.

  • Takeaways & Limitations

    The method prefers clear target subjects and simpler backgrounds; people in the background make subjects ambiguous.

Abstract

from arXiv · show

We introduce a robust, real-time, high-resolution human video matting method that achieves new state-of-the-art performance. Our method is much lighter than previous approaches and can process 4K at 76 FPS and HD at 104 FPS on an Nvidia GTX 1080Ti GPU. Unlike most existing methods that perform video matting frame-by-frame as independent images, our method uses a recurrent architecture to exploit temporal information in videos and achieves significant improvements in temporal coherence and matting quality. Furthermore, we propose a novel training strategy that enforces our network on both matting and segmentation objectives. This significantly improves our model's robustness. Our method does not require any auxiliary inputs such as a trimap or a pre-captured background image, so it can be widely applied to existing human matting applications.

1. Introduction

The paper targets robust, real-time human video matting without auxiliary inputs by exploiting temporal information and jointly training matting and segmentation objectives. It reports a lighter, faster method for high-resolution video.

  • Real-time background replacement for human subjects is needed in applications such as video conferencing and entertainment video creation.
  • Current neural matting solutions can lack robustness and generate artifacts, motivating improvements in matting quality and robustness.
  • Frame-independent methods neglect temporal information, which can reduce flicker, improve perceptual quality, and help resolve ambiguous boundaries using previous frames.
  • The proposed training strategy jointly enforces matting and semantic segmentation objectives.
  • 58% parameters and 4K 76 FPS plus HD 104 FPS are reported for the method on an Nvidia GTX 1080Ti GPU.

2. Related Works

Related work spans trimap-based, background-based, auxiliary-free, video-native, recurrent, and high-resolution matting approaches, each providing different guidance or efficiency trade-offs.

  • Trimap-based methods require manual trimap annotations and are often object agnostic, making them suitable for interactive photo editing.
  • Background matting uses a pre-captured background to improve accuracy but cannot handle dynamic backgrounds and large camera movements.
  • Semantic segmentation predicts pixel-level class labels without auxiliary inputs, but directly using its binary mask for replacement produces strong artifacts.
  • Auxiliary-free matting methods cover general objects or human portraits, whereas MODNet targets portraits and this work targets the full human body.
  • Existing video approaches provide post-processing, short-term cues, or trimap propagation, while recurrent architectures are adopted here for video matting.
  • High-resolution matting uses selective patch refinement or guided-filter upsampling; Deep Guided Filter is learnable end-to-end and chosen for speed.

3. Model Architecture

The architecture combines a frame-wise encoder, multi-scale recurrent decoder, and optional Deep Guided Filter to produce high-resolution alpha and foreground predictions while aggregating temporal information.

  • The network uses an encoder for frame features, a recurrent decoder for temporal aggregation, and a Deep Guided Filter for high-resolution upsampling.
  • Feature-Extraction Encoder: The encoder follows semantic-segmentation designs, using MobileNetV3-Large and LR-ASPP to efficiently locate human subjects.
  • Recurrent Decoder: ConvGRU aggregates temporal information at multiple scales and is selected for greater parameter efficiency than ConvLSTM.
  • Recurrent Decoder: The recurrent hidden state serves as both the current output and the recurrent state for the next time step, starting from an all-zero tensor.
  • Recurrent Decoder: ConvGRU operates on half the channels through split and concatenation, reducing computation while preserving a branch for current-frame spatial features.
  • Recurrent Decoder: The output block projects features to alpha, foreground, and segmentation predictions without using ConvGRU at that scale.
  • High-Resolution Upsampling: For high-resolution input, the encoder-decoder processes a downsampled frame and DGF combines low-resolution outputs with the high-resolution frame to produce refined predictions.

4. Training

The training strategy jointly optimizes matting and human segmentation, using staged exposure to longer sequences and higher resolutions plus diverse real and synthetic data. It combines matting, temporal-coherence, foreground, and segmentation losses across frames.

  • Training objectives: Joint matting and semantic-segmentation training uses real segmentation data to reduce overfitting to synthetic matting distributions.The authors motivate this by the similarity between human matting and segmentation and the greater availability of segmentation data.
  • Datasets and augmentation: The training data combines VideoMatte240K, Distinctions-646, Adobe Image Matting, YouTubeVIS, COCO, and SPD, with human-focused image and video samples.VideoMatte240K contributes 484 4K/HD clips; segmentation data includes 2985 YouTubeVIS clips, 64,111 COCO images, and 5711 SPD samples.
  • Datasets and augmentation: Motion and temporal augmentations continuously vary appearance and geometry across foregrounds and backgrounds to increase training data variety.Augmentations include translation, scale, rotation, photometric changes, noise, blur, and nonlinear temporal changes.
  • Training procedure: The model is trained progressively on longer sequences and higher resolutions through four stages, balancing low-resolution temporal context with high-resolution processing.Stage 2 increases sequence length to T = 50, while Stage 3 combines low-resolution long sequences with high-resolution short sequences using DGF.
  • Training procedure: Segmentation training alternates with matting iterations, using video segmentation data on even iterations and image segmentation data as one-frame sequences on odd iterations.This interleaving is applied throughout all training stages and follows each stage’s batch, sequence, and resolution settings for video data.
  • Losses: The objective sums alpha reconstruction and Laplacian losses, alpha temporal coherence, foreground L1 and temporal coherence losses, while segmentation uses binary cross entropy.All losses are applied over frames t ∈ [1, T].

5. Experimental Evaluation

The evaluation compares matting quality, temporal coherence, robustness, model size, and speed across composited and real-video settings. The method consistently improves alpha quality and coherence while remaining lightweight and real-time at high resolution.

  • Evaluation Setup: The benchmark composites 100-frame clips from VM, D646, and AIM onto five video and five image backgrounds, with motion augmentation for image samples.
  • Evaluation Setup: The evaluation measures alpha quality with MAD, MSE, Grad, and Conn, temporal coherence with dtSSD, and foreground quality with MSE on pixels where α∗>0.MAD and MSE are scaled by 1e3, while dtSSD is scaled by 1e2 for readability.
  • Composition Datasets: Ours predicts more accurate and consistent low-resolution alpha than all compared methods across datasets, although BGMv2 has better foreground prediction.The comparisons include FBA with synthetic trimaps, BGMv2 with the first frame’s ground-truth background, and MODNet with neighbor-frame smoothing.
  • Composition Datasets: Ours outperforms MODNet with Fast Guided Filter on all high-resolution alpha metrics.The high-resolution comparison removes Conn because it is too expensive to compute at that resolution.
  • Real Videos: On real videos, ours preserves fine-grained details, resists semantic errors, and handles fast-moving body parts better than MODNet.The qualitative comparisons cover real, cellphone, webcam, and random YouTube videos.
  • Size and Speed Comparison: Ours uses only 58% of MODNet’s parameters and reaches 104 FPS at HD and 76 FPS at 4K on an Nvidia GTX 1080Ti.It is fastest on HD, while being slightly slower than some alternatives at lower resolution and 4K; GMACs is only a rough approximation.

6. Ablation Studies

The ablations show that temporal recurrence improves matting quality and consistency, while segmentation training improves robustness. Additional evaluations characterize performance across backgrounds, model variants, and challenging scenes.

  • 6.1. Role of Temporal Information: The error drops significantly during the first 15 frames and then remains stable, whereas MODNet shows large metric fluctuations.Disabling recurrence worsens quality and consistency, supporting the role of temporal information.
  • 6.1. Role of Temporal Information: Our method produces more detailed alpha predictions and handles semantic errors and fast-moving body parts better than the compared methods.Qualitative comparisons include YouTube, cellphone, and webcam videos.
  • 6.1. Role of Temporal Information: The recurrent hidden state reconstructs revealed background regions, tracks motion history, and forgets useless memory after shot cuts.These learned recurrent channels support future predictions, including under camera movement.
  • 6.2. Role of Segmentation Training Objective: 61.50 mIOU is achieved on the human category of COCO, while removing the segmentation objective reduces performance to 38.24 mIOU.Thresholding alpha at 0.5 still achieves 60.88 mIOU.
  • 6.3. High-Resolution Upsampling: DGF improves Grad and dtSSD over FGF with only a small size and speed overhead, while MAD and MSE remain inconclusive.Grad reflects high-resolution detail accuracy, and dtSSD reflects temporal coherence.
  • 6.4. Static vs. Dynamic Backgrounds: The method handles static and dynamic backgrounds, performs slightly better on static backgrounds, and BGMv2 performs poorly on dynamic backgrounds.BGMv2’s static-background advantage depends on ground-truth background alignment and may degrade with misalignment.
  • 6.5. Model Variants: The larger ResNet50 model with more decoder channels improves performance and is more suitable for server-side applications.The default and large variants use different backbones and decoder capacities.
  • 6.6. Limitations: Performance is less reliable when people in the background make the target ambiguous or when scenes contain complex backgrounds.Clear target subjects and simpler backgrounds are preferred.

7. Conclusion

The paper concludes that recurrent temporal modeling and joint matting-segmentation training produce robust human video matting. It reports state-of-the-art performance with a lighter, faster model and emphasizes temporal information’s role in quality and consistency.

  • 7. Conclusion: The proposed recurrent architecture achieves state-of-the-art human video matting while being lighter and faster.The conclusion presents the architecture as the paper’s central contribution.
  • 7. Conclusion: Temporal information plays an important role in improving matting quality and consistency.The recurrent design exploits temporal information throughout video processing.
  • 7. Conclusion: Joint training on matting and semantic segmentation objectives improves robustness across various types of videos.The conclusion identifies this training strategy as a second major contribution.

B. Network

The network combines a frame-wise encoder, recurrent decoder, and Deep Guided Filter for high-resolution human matting. Its outputs include foreground, alpha, and segmentation predictions.

  • B. Network: The default network uses a MobileNetV3-Large backbone, while the large variant uses ResNet50.Table 9 describes feature-channel variants across scales.
  • B. Network: The encoder extracts feature maps at 1/2, 1/4, 1/8, and 1/16 scales, using dilated convolutions without further downsampling in its last block.The 1/16 feature map is passed to LR-ASPP.
  • B. Network: The recurrent decoder uses ConvGRU layers and aggregates temporal information through recurrent hidden states.Decoder channels are split and concatenated around ConvGRU operations.
  • B. Network: The final projection outputs five channels: three-channel foreground, one-channel alpha, and one-channel segmentation predictions.The decoder uses convolutional blocks before the final projection.
  • B. Network: DGF takes predicted foreground, alpha, and final hidden features as inputs for high-resolution processing.Its internal convolutions use 1 × 1 kernels and 16 filters.

C. Training

The training procedure uses staged video and image segmentation passes with randomized sequence lengths, batch sizes, resolutions, and mixed-precision multi-GPU training.

  • C. Training: Algorithm 1 alternates video and image segmentation passes as part of the proposed training strategy.The procedure includes VideoSegmentationPass and ImageSegmentationPass stages.
  • C. Training: The training uses batch size B = 4, with B′ = B × T, and randomly samples resolutions from 256–512 and 1024–2048 ranges.Sequence lengths T and T̂ vary according to the training stages.
  • C. Training: Training runs on 4 Nvidia V100 32G GPUs with mixed precision to reduce GPU memory consumption.The reported stage durations are approximately 18, 2, 8, and 14 hours.

D. Data Samples

The data samples combine natural and artificially generated motion for training, while testing uses weaker affine motion augmentation applied only to foregrounds and backgrounds.

  • Training Samples: Training composites include natural movements from video compositing and artificial movements generated through motion augmentation.
  • Training Samples: The final column visualizes motion by showing each pixel’s standard deviation across time.
  • Testing Samples: Testing composites apply motion augmentation only to image foregrounds and backgrounds.
  • Testing Samples: Testing augmentation consists of affine transforms and is weaker than training augmentation to keep samples realistic.

E. Additional Results

Additional comparisons show that the recurrent model is more robust and temporally coherent than MODNet, while its hidden state stores useful temporal information and forgets it after shot cuts.

  • Additional Qualitative Results: Our method is consistently more robust than MODNet in additional qualitative comparisons.
  • Temporal Coherence: MODNet flickers in low-confidence regions, whereas our results remain temporally coherent.
  • Recurrent Hidden States: The recurrent hidden state stores useful temporal information and forgets useless information upon shot cuts.
Loading 2108.11515v1…