Source-linked AI summary
Background Matting: The World is Your Green Screen
Soumyadip Sengupta, Vivek Jayaram, Brian Curless, Steve Seitz, Ira Kemelmacher-Shlizerman
TL;DR
High-quality person matting typically requires a green screen or manually created trimap, motivating a less laborious alternative for casual capture. The paper uses an additional background photo and a deep, cue-combining network with supervised synthetic training and self-supervised adversarial adaptation. It reports improved performance over competing methods, including lower Adobe-dataset errors and significantly better user-study results, within static-background, small-motion settings.
Problem
High-quality person matting generally requires a green screen or painstaking manual trimap, while existing automatic trimap-free methods are not comparably effective.
Method
The method combines an input image, an additional background photo, soft segmentation, and optional video cues in a deep network trained on synthetic composites and adapted to real imagery.
Results
The method improves over competing approaches, achieving SAD 1.72 and MSE 0.97e−2 versus LFM’s SAD 2.00 and MSE 1.08e−2, while significantly outperforming alternatives in user studies.
Takeaways & Limitations
Casually captured background images can support high-quality person mattes without green screens or painstaking detailed trimaps.
Takeaways & Limitations
The approach requires a static background, small camera motion, and one or more people as foregrounds.
Abstract
from arXiv · showhide
We propose a method for creating a matte -- the per-pixel foreground color and alpha -- of a person by taking photos or videos in an everyday setting with a handheld camera. Most existing matting methods require a green screen background or a manually created trimap to produce a good matte. Automatic, trimap-free methods are appearing, but are not of comparable quality. In our trimap free approach, we ask the user to take an additional photo of the background without the subject at the time of capture. This step requires a small amount of foresight but is far less time-consuming than creating a trimap. We train a deep network with an adversarial loss to predict the matte. We first train a matting network with supervised loss on ground truth data with synthetic composites. To bridge the domain gap to real imagery with no labeling, we train another matting network guided by the first network and by a discriminator that judges the quality of composites. We demonstrate results on a wide variety of photos and videos and show significant improvement over the state of the art.
1. Introduction
The paper introduces trimap-free background matting for people using an additional photo of the static background, avoiding green screens and laborious manual trimaps. A deep network combines image, background, segmentation, and optional temporal cues, with training designed to improve performance on real imagery.
- Motivation: An extra background photo enables trimap-free person matting in everyday settings, requiring far less effort than manually painting a trimap.The approach assumes the subject and camera move only slightly between the two captures.
- Motivation: Background matting remains underconstrained because each pixel has four unknown foreground-and-alpha values but only three color observations.Shadows, reflections, similar foreground-background colors, and imperfect background captures further complicate estimation.
- Approach: The network estimates foreground and alpha from the original image, background photo, automatically computed soft segmentation, and optional nearby video frames.Most comparisons to single-image methods use no temporal cues.
- Approach: A Context Switching Block selectively combines input cues to reduce the synthetic-to-real domain gap, alongside dataset augmentation and additional training.The method also uses self-supervised adversarial training on real images, as summarized among the contributions.
- Limitations: The method requires two images, a static background, small camera motion, and human foregrounds, limiting its scope despite strong results in that scenario.Moving people in the background or large camera motion can cause the method to perform poorly.
2. Related Work
Prior natural-image matting methods commonly rely on trimaps, while newer trimap-free human methods use segmentation cues. This work instead uses a casually captured natural background and combines multiple cues for human matting.
- Traditional approaches: Natural-image matting traditionally uses trimaps, with sampling-based methods estimating color statistics and propagation-based methods spreading alpha into unknown regions.These approaches represent the main non-learning-based categories discussed by the paper.
- Learning-based approaches: Deep learning improved natural-image matting, particularly with user-generated trimaps, and later work used adversarial loss to improve predictions from synthetic composites.The Adobe Matting Dataset provides real mattes composited over varied backgrounds for training.
- Trimap-free approaches: Trimap-free human matting methods use semantic segmentation cues and often predict a trimap before predicting alpha.Related methods address portraits and whole bodies without manually supplied trimaps.
- Known-background matting: Background subtraction methods with known natural backgrounds are sensitive to thresholds, often produce binary mattes, or treat shadows as foreground.The paper distinguishes its approach from these simpler background-difference techniques.
- Video matting: Video matting methods have used known backgrounds, optical flow, and temporal smoothing, but typically still require trimaps for keyframes.The paper’s overview also includes motion cues for video input.
3. Our Approach
The approach combines a casually captured background, soft person segmentation, and optional temporal cues in a deep matting network. Context Switching features and self-supervised adversarial training address failures when synthetic training data differs from real imagery.
- Inputs: The system takes a person image or video, a separately captured static background, and a soft subject segmentation as inputs.Nearby video frames can provide additional motion cues; handheld capture assumes small camera motion and aligns the background with a homography.
- Training: The network is first trained with supervised losses on synthetically composited Adobe Matting data, using non-transparent foreground objects and augmented backgrounds.The supervised objective includes foreground and alpha terms, with an alpha-gradient term encouraging sharper mattes.
- Training: Self-supervised adversarial training jointly refines a real-image network and discriminator using teacher outputs, novel-background composites, and real inputs.The Adobe-trained network supplies pseudo-ground truth, while the discriminator judges whether composites appear realistic; initialization and a small teacher loss help avoid the degenerate alpha = 1 solution.
- Matting network: The matting network G predicts foreground color and alpha from the image, registered background, soft segmentation, and optional nearby frames.For video, the temporal input uses nearby grayscale frames; without video, repeated grayscale copies of the input are used.
- Domain gap: A conventional concatenation network trained on synthetic composites over-relied on the background and produced holes when foreground colors matched it.These failures motivated architecture changes aimed at using segmentation cues more effectively and generalizing to real data.
- Matting network: The Context Switching block selectively combines image, background, segmentation, and motion features conditioned on the input image.Separate encoders and selector blocks combine each auxiliary cue with image features before the rest of the encoder-decoder network.
4. Experimental Evaluation
The method is evaluated on synthetic composites and real handheld or fixed-camera imagery using comparisons, user studies, and qualitative examples. It generally outperforms alternatives, while registration errors and dynamic backgrounds remain challenging.
- The evaluation compares the approach with Bayesian, Context-Aware, and Index Matting, among other alternatives.
- Results on Synthetic-Composite Adobe Dataset: 220 synthetic composites from 11 held-out human mattes and 20 random backgrounds are used for quantitative comparison.
- Results on Synthetic-Composite Adobe Dataset: The approach is more robust to background perturbation than BM and improves on CAM, IM, and other trimap-based algorithms.
- Results on Real Data: The real-data evaluation uses handheld and fixed-camera smartphone videos, with a separate background capture and no temporal information for comparisons to prior methods.
- Results on Real Data: User studies on real videos show significant overall improvement over alternatives, with larger gains for fixed-camera results.
- Results on Real Data: Single-image examples show improvement over alternatives, while a dynamic-background fountain is presented as a failure case.
5. Ablation Studies
Ablations examine motion cues, Context Switching, and real-data adversarial training. These components improve performance in real imagery, especially when foreground and background colors are similar.
- Motion cues help produce cleaner alpha mattes when foreground color matches the background or the foreground moves substantially.
- On the synthetic Adobe dataset, Ours-Adobe achieves a SAD score of 1.73 versus 3.50 for Ours-Real.
- On real data, Ours-Real significantly outperforms Ours-Adobe in qualitative examples and an additional user study.
- The Context Switching Block uses segmentation, color-difference, and available motion cues more effectively than naive concatenation, which can generate holes when colors are similar.
6. Conclusion
The paper presents background matting for casual capture using subject and empty-background images, avoiding green screens and detailed trimaps. It combines synthetic-composite training with adversarial adaptation to real data.
- The technique captures high-quality foreground-and-alpha mattes in natural settings using images with and without a human subject.
- The approach avoids green screens and painstaking detailed trimaps by requiring the subject to remain nearly still between two shots.
- A deep learning framework is trained on synthetic composites and adapted to real data using an adversarial network.
- The appendix provides additional network-architecture details, automatic-trimap analysis, foreground-layer analysis, and qualitative ablation results.
B. Network Architectures
The matting network uses a Context Switching Block, residual blocks, and separate decoders to predict alpha and foreground. Its design also incorporates compositing quality considerations and jointly estimates foreground and alpha.
- Network Architecture: The network predicts the alpha matte and foreground layer through separate decoder branches after shared Context Switching and residual blocks.
- Context Switching Block: The Context Switching Block combines image features with prior features to improve matting by selecting among input cues.
- Alpha Decoder: The alpha decoder upsamples features and produces alpha values scaled between −1 and 1 by a final Tanh layer.
- Foreground Decoder: The foreground decoder includes a skip connection from image features before producing the foreground layer.
- Discriminator: A 70 × 70 PatchGAN discriminator evaluates patches across the composite image to produce real-or-fake judgments.
- Compositing: Jointly estimating foreground and alpha produces fewer compositing artifacts than deriving the foreground in baseline methods that predict only alpha.
C. Experimental Evaluation
The evaluation examines automatic trimap generation and foreground estimation for compositing. Probability-thresholded segmentation is favored over background-subtraction and morphology-based alternatives, while direct foreground prediction avoids artifacts from solving with a perturbed background.
- Automatic Trimap Generation: Automatic trimaps label segmentation probabilities above 0.95 as foreground, below 0.05 as background, and intermediate values as unknown.
- Automatic Trimap Generation: Background subtraction is unsuitable as a direct matte because it is highly sensitive to color differences and produces only binary output.
- Automatic Trimap Generation: Probability-thresholded automatic trimaps outperform background subtraction and erosion-dilation of the segmentation mask.
- Predicting Foreground layer F: Estimating foreground from the matting equation using captured background B′ performs worse than setting F = I because handheld misalignment creates discrepancies between B′ and B.
- Predicting Foreground layer F: The matting equation leaves infinitely many foreground and background pairs consistent with observed I and alpha, making foreground recovery non-trivial when alpha alone is predicted.
C.3. Results on Real Data
The real-data evaluation includes fixed-camera and handheld user studies, with an updated analysis addressing connected-component post-processing. The reported results remained similar after correcting the study procedure.
- Study Correction: The original user study favored the proposed method by retaining only its largest alpha-connected component, unlike for competing approaches.
- Study Correction: CAM and IM were rerun with connected-component removal after small floaters were observed, while BM did not show comparable floater artifacts.
- Study Correction: LFM was left without this post-process because it could absorb background regions larger than the person and remove the foreground subject.
- Results: The updated user-study results were similar to the main-paper results, and excess connected components in CAM and IM had no significant impact relative to other matte-estimation errors.
- User Study: The user studies evaluate 10 real-world videos with fixed cameras and 10 with handheld cameras.
- Additional Analyses: The appendix also analyzes the Context Switching Block, motion cues, and the comparison between Ours-Real and Ours-Adobe.
D.1. Role of Context Switching Block
The Context Switching Block helps the network use soft segmentation when foreground and background colors coincide, avoiding holes produced by relying mainly on color differences.
- D.1. Role of Context Switching Block: No Context Switching produces holes when the foreground color strongly matches the background.
- D.1. Role of Context Switching Block: The ablation compares Ours-Adobe with Context Switching against Gconcat without the Context Switching Block on real video.Both networks use the same motion-cue input in this experiment.
- D.1. Role of Context Switching Block: The Context Switching Block improves cue selection by using soft segmentation when foreground and background colors coincide.Without context switching, the network focuses too heavily on color differences and fails in these cases.
- D.1. Role of Context Switching Block: The Context Switching Block handles color coincidences better across multiple frames than the no-context-switching model.The no-context-switching model repeatedly produces holes when colors coincide significantly.
D.2. Role of Motion Cues
Motion cues improve alpha-matte estimation when foreground colors match the background or the foreground moves substantially, but their usefulness is limited by ambiguity, camera motion, and short temporal windows.
- D.2. Role of Motion Cues: For 60fps video, the motion cue uses four frames at offsets −2T, −T, +T, and +2T with T=20 frames.
- D.2. Role of Motion Cues: Motion cues improve alpha-matte estimation when foreground colors match the background and the foreground moves significantly.Additional frames reveal regions that move and are therefore more likely to belong to the foreground.
- D.2. Role of Motion Cues: Motion is not always a reliable foreground indicator because cast shadows can move with the foreground.
- D.2. Role of Motion Cues: Small handheld-camera motion can create background motion through misregistration, while a 1.33-second window may lack useful motion information.
- D.2. Role of Motion Cues: Comparisons with competing methods disable temporal information by setting the motion cue to M = {I, I, I, I}.Images, including video frames, are treated independently for those comparisons.
D.4. Role of background and segmentation
The captured background image is more important than soft segmentation for performance: removing it sharply worsens synthetic-data error, while segmentation perturbations have little effect.
- D.4. Role of background and segmentation: 8.33 SAD error without the background image versus 1.73 with it on the synthetic-composite Adobe dataset.
- D.4. Role of background and segmentation: The captured background image is more crucial to performance than the estimated soft segmentation.
- D.4. Role of background and segmentation: Eroding and dilating the segmentation by 20 steps increases SAD error only from 1.73 to 1.76 on the synthetic-composite Adobe dataset.This indicates robustness to segmentation errors.
- D.4. Role of background and segmentation: Ours-Real is generally better than Ours-Adobe, although some examples produce an inferior matte.