Source-linked AI summary
Real-Time High-Resolution Background Matting
Shanchuan Lin, Andrey Ryabtsev, Soumyadip Sengupta, Brian Curless, Steve Seitz, Ira Kemelmacher-Shlizerman
TL;DR
High-resolution real-time matting must preserve fine details while avoiding the manual inputs and computational limits of earlier approaches. The paper combines background matting, multi-stage training data, and selective refinement, achieving real-time 4K and HD operation with state-of-the-art background matting results.
Problem
Existing tools produce fine-detail boundary artifacts, while earlier high-quality methods lack real-time high-resolution operation or require manual input.
Method
The method uses an extra background image, staged datasets, and a low-resolution base network followed by selective high-resolution refinement.
Results
The technique achieves state-of-the-art background matting at 4K 30fps and HD 60fps.
Takeaways & Limitations
Selective high-resolution refinement reduces redundant computation, making real-time high-resolution matting practical for real-world videos and images.
Takeaways & Limitations
Handheld input is limited to small motion, and best results require a simple-textured background, fixed exposure, focus, white balance, and a tripod.
Abstract
from arXiv · showhide
We introduce a real-time, high-resolution background replacement technique which operates at 30fps in 4K resolution, and 60fps for HD on a modern GPU. Our technique is based on background matting, where an additional frame of the background is captured and used in recovering the alpha matte and the foreground layer. The main challenge is to compute a high-quality alpha matte, preserving strand-level hair details, while processing high-resolution images in real-time. To achieve this goal, we employ two neural networks; a base network computes a low-resolution result which is refined by a second network operating at high-resolution on selective patches. We introduce two largescale video and image matting datasets: VideoMatte240K and PhotoMatte13K/85. Our approach yields higher quality results compared to the previous state-of-the-art in background matting, while simultaneously yielding a dramatic boost in both speed and resolution.
1. Introduction
The paper addresses artifacts and scalability barriers in automated background replacement by combining background matting with selective high-resolution refinement. The resulting technique targets strand-level detail while operating in real time at 4K and HD.
- Background replacement tools produce boundary artifacts around fine details such as hair and glasses.
- Traditional image matting provides higher quality but is not real-time at high resolution and often requires manual input.
- The method captures an additional background image to recover the alpha matte and foreground layer.
- The technique produces state-of-the-art background matting at 4K 30fps and HD 60fps.
- The previous state-of-the-art background matting method is limited to 512×512 resolution at 8fps.
- A base network predicts low-resolution outputs and an error map, while a refinement network produces high-resolution output only on selected regions.
2. Related Work
Prior background-replacement methods use segmentation, trimaps, no external input, or known backgrounds, each with distinct quality, automation, or resolution constraints. The paper positions its approach within known-background matting while adopting learned selective refinement.
- Segmentation: Segmentation is efficient but tends to produce coarse boundaries, especially at higher resolutions.
- Segmentation: PointRend improves large-resolution segmentation by sampling and iteratively refining points near boundaries.
- Trimap-based matting: Traditional trimap-based matting requires manual trimap annotation and solves for alpha in the unknown region.
- Matting without any external input: Methods without external input directly predict alpha mattes, with portrait matting benefiting from lower variability than full-body human images.
- Matting with a known natural background: Background Matting uses an additional captured background to predict alpha and foreground but is limited to 512×512 at 8fps.
3. Our Dataset
The paper introduces large-scale video and photo matting datasets to address the limited quantity and variation of existing high-quality data. VideoMatte240K emphasizes diverse continuous video frames, while PhotoMatte13K/85 emphasizes high-resolution detail.
- Existing public datasets provide too few samples to learn large variations in human poses and fine details at high resolution.
- The paper introduces two additional datasets containing 240k unique frames and 13k unique photos.
- VideoMatte240K: VideoMatte240K contains 240,709 unique frames from 484 high-resolution green-screen videos, including 4K and HD footage.
- VideoMatte240K: VideoMatte240K provides diverse human subjects, clothing, and poses, and is the first public video matting dataset with continuous frame sequences.
- PhotoMatte13K/85: PhotoMatte13K contains 13,665 high-resolution studio images with manually tuned and repaired mattes, including individual hair strands.
- PhotoMatte13K/85: PhotoMatte85 is a released 85-matte test set created because privacy and licensing issues prevent sharing PhotoMatte13K.
4. Our Approach
The approach combines low-resolution whole-image prediction with selective high-resolution refinement to make background matting efficient while preserving detailed regions. It also uses residual foreground prediction, error-guided patch selection, and staged multi-dataset training.
- Background Matting: A background image and input image are used to predict alpha and foreground, enabling compositing onto any new background.The model predicts a foreground residual and recovers the foreground by adding it to the input with clamping.
- Base Network: The base network processes downsampled inputs to produce coarse alpha, foreground-residual, error-map, and hidden-feature predictions.Its encoder-decoder design is inspired by DeepLabV3 and DeepLabV3+ and includes a backbone, ASPP, and decoder.
- Refinement Network: The refinement network operates only on regions selected by the predicted error map, producing full-resolution alpha and foreground residual outputs.The architecture directly upsamples the coarse result elsewhere, while selected patches undergo two-stage refinement.
- Refinement Network: The method selects the top k error locations after resampling the error map so each selected location corresponds to a 4×4 original-resolution patch.This yields 16k refined pixels at the original resolution, reducing computation relative to refining the entire image.
- Training: The training objectives supervise alpha, foreground, and error-map prediction, with the alpha loss also penalizing Sobel-gradient differences.The error-map target is the absolute alpha error, encouraging high predicted error in complex regions such as hair.
5. Experimental Evaluation
The evaluation compares the proposed method with trimap-based, background-based, and chroma-keying approaches on benchmark and real-world data. It reports stronger high-resolution quality than BGM, practical real-time performance, and advantages under automatic or unevenly lit capture conditions.
- Composition datasets: The method is evaluated against DIM, FBA, BGM, and an adapted BGM model across composited datasets using alpha and foreground quality metrics.The benchmarks include samples from AIM, Distinctions, and PhotoMatte85 composited onto multiple backgrounds with simulated capture imperfections.
- Composition datasets: It outperforms the existing background-based BGM method across all evaluated datasets.
- Real images: The method produces sharper, more detailed hair and edge results than competing methods at high resolution, while FBA remains strong with manual trimaps but is slower and resolution-limited.Automatic trimaps can cause large FBA artifacts when segmentation is faulty.
- Captured data: 59% of user-study judgments preferred the proposed method to BGM, compared with 23% for BGM; for sharp 4K-or-larger samples, preferences were 75% versus 15%.The study used 40 participants and 34 frames from captured and shared test videos and photos.
- Performance comparison: The method reaches HD 60fps and 4K 30fps on an Nvidia RTX 2080 TI with batch size 1, using 55.7% of BGM’s parameters.A MobileNetV2 variant reaches 4K 45fps and HD 100fps, while BGM handles 512×512 at 7.8fps.
- Practical use: Under uneven amateur green-screen lighting, the method outperforms approaches designed for green screens.
6. Ablation Studies
The ablations examine dataset composition, backbone choice, selective refinement, and refinement kernels. They show that training data and patch-based refinement improve quality, while backbone and refinement-area choices expose speed–quality trade-offs.
- Role of datasets: Removing VideoMatte240K, PhotoMatte13K, or Distinctions worsens metrics, whereas appending AIM as a fourth training stage worsens metrics even on AIM.The authors attribute the AIM degradation to lower resolution and quality and possible overfitting from its small sample count.
- Role of the base network: ResNet-50 is often sufficient for best quality; MobileNetV2 is worse on all metrics but is significantly faster and smaller while still outperforming BGM’s metrics.ResNet-101 improves some metrics but worsens others relative to ResNet-50.
- Role of the refinement network: Most improvement is achieved by refining only 5% to 10% of the image area, while selected-patch refinement is faster than refining the full image.
- Role of the refinement network: Refinement improves detail sharpness and remains effective at 4K resolution.
- Patch-based versus point-based refinement: The 3×3 refinement kernel achieves better metrics than point-based 1×1 kernels because it provides a larger receptive field.The 3×3 stack yields a 13×13 receptive field for each output pixel, compared with 2×2 for the point-based alternative.
- Limitations: For best results, the method is recommended with a simple-textured background, fixed exposure, focus, and white balance, and a tripod; handheld alignment is limited to small motion.
7. Conclusion
The method enables real-time, high-resolution background replacement and reports practical deployment in Zoom, while acknowledging potential misuse of image editing.
- The technique operates at 4K resolution at 30fps and HD resolution at 60fps.
- The system streams results to Zoom, producing a more realistic virtual conference call.
- Image editing enabled by the method could be used negatively, motivating watermarking and other security techniques in commercial applications.
B.1. Architecture
The architecture combines modified convolutional backbones with a coarse-to-fine refinement pipeline that processes only selected high-resolution patches, using vectorized operations for performance.
- Backbone: The backbone uses modified ResNet or MobileNetV2 networks with six-channel inputs and output stride 16.The six channels accommodate the input and background images.
- Base network: The base network outputs coarse alpha, foreground residual, error-map, and hidden-feature channels.Its 37-channel output contains one alpha channel, three foreground-residual channels, one error-map channel, and 32 hidden-feature channels.
- Refinement network: The refinement network crops the top k most error-prone patches and combines them with corresponding input and background patches.Patch refinement uses nearest upsampling before concatenating the selected patches with image and background information.
- Output formation: The refined patches replace corresponding locations after coarse outputs are bilinearly upsampled to full resolution.
- Implementation: Vectorized patch extraction and replacement, together with faster nearest upsampling on small patches, supports efficient implementation.
C. Dataset
The paper introduces large-scale matting datasets spanning video and high-resolution image data, with varied backgrounds and training configurations documented for the final model.
- VideoMatte240K: VideoMatte240K contains 484 video clips totaling 240,709 frames.Clips average 497.3 frames, with lengths ranging from 124 to 1500 frames.
- Backgrounds: Background images are crawled using keywords covering locations such as airports, beaches, offices, forests, and workplaces.
- Training setup: The final model uses one RTX 2080 Ti for base-network training and two RTX 2080 Ti GPUs for joint training.Training order, epochs, and hours are recorded for the different datasets.
D.1. Training augmentation
Training augmentation combines compositing, geometric and photometric perturbations, simulated misalignment, and artificial shadows to improve robustness to real-life captures.
- Compositing: Foreground and background samples are paired in a rotating zip fashion to form composite training epochs.The rotation accommodates datasets with different sizes by reusing samples until one set is exhausted.
- Image augmentation: Each sample independently augments foreground and background with geometric, photometric, noise, blur, and sharpening transformations.The input image is composited as I = αF + (1 − α)B.
- Misalignment: Background-only perturbations introduce small input-background misalignments to improve robustness on real-life captures.The background is altered through limited rotation, translation, brightness, contrast, saturation, and hue changes.
- Shadow augmentation: Artificial shadows are added behind subjects 30% of the time because real subjects often cast shadows on their surroundings.
- Evaluation setup: Evaluation pairs human test samples with random backgrounds and uses the method and metrics described in prior work.AIM and Distinctions each yield 55 images, while PhotoMatte85 yields 85 images.
- Performance evaluation: The experiments compare performance on RTX 2080 Ti and RTX 2060 Super GPUs, with batch size and precision affecting FPS.The entry-level GPU produces lower FPS but remains within an acceptable range for many real-time applications.
F. Additional Results
The additional results include a user-study interface, qualitative comparisons across 34 examples, and performance measurements across GPUs, batch sizes, and precisions.
- 34 examples are presented with average user ratings and results from different methods.
- The user study compares Ours and BGM on original images and two result images, with ratings from “much better” to “similar”.
- Performance is reported across different GPUs, batch sizes, and numerical precisions.
- The additional results include three sets of qualitative comparisons accompanied by average ratings.