Source-linked AI summary
αDepth: Learning Single-Pass Soft Boundary Decomposition for Stereo Conversion
Xiang Zhang, Yang Zhang, Lukas Mehl, Karlis Martins Briedis, Markus Gross, Christopher Schroers
TL;DR
Stereo conversion struggles at soft boundaries because mixed foreground-background pixels create ambiguous depth and existing matting methods often require guidance or fail in complex scenes. αDepth jointly predicts layered color, depth, and opacity using CAR for local, automatic decomposition. Experiments report state-of-the-art stereo-conversion boundary fidelity and reduced boundary artifacts.
Problem
Soft boundaries mix foreground and background colors while conventional depth models assign single-layer depth; existing matting methods may require guidance and struggle with complex multi-target scenes.
Method
αDepth jointly estimates layered foreground-background color and depth, using CAR to encode local opacity continuously and infer scene-level decompositions in one forward pass.
Results
αDepth consistently outperforms previous stereo-conversion methods, eliminates artifacts such as background bleeding, and achieves state-of-the-art boundary fidelity.
Takeaways & Limitations
CAR enables automatic scene-level soft-boundary decomposition without manual guidance, including in complex multi-target scenes.
Takeaways & Limitations
Global scene structure and scale remain dependent on the quality of the initial depth map, so severe base-geometry errors may persist.
Abstract
from arXiv · showhide
Accurately modeling soft boundaries, e.g., hair and defocus blur, is a fundamental challenge in stereo conversion due to the ambiguous blending of foreground and background. Existing depth models primarily predict single-layer depth, leading to ambiguity in depth correspondence at soft boundaries. While matting techniques can capture opacity for layered modeling, they often struggle in complex scenes with multiple targets and usually require user intervention. This paper introduces αDepth, a layered representation that decomposes soft boundaries for high-fidelity stereo conversion. Specifically, we first resolve mixed color and depth ambiguity by estimating layered color and depth values at soft boundaries. Considering complex multi-target scenes, we design a Circular Alpha Representation (CAR) that shifts the paradigm from global target extraction to local boundary decomposition. Unlike prior matting methods restricted to a single foreground/background, CAR enables efficient scene-level inference without manual guidance. Extensive evaluations demonstrate that αDepth achieves state-of-the-art performance in stereo conversion, eliminating background bleeding and structural distortions at soft boundaries.
1 Introduction
αDepth addresses soft-boundary ambiguity in stereo conversion by decomposing mixed foreground and background into layered color and depth. CAR enables local, scene-level boundary modeling without manual guidance.
- Soft boundaries mix foreground and background colors within pixels, creating ambiguous depth correspondence and artifacts such as halos, bleeding, and floating textures.
- Conventional depth estimators assign one depth per pixel, failing to represent layered soft boundaries and causing broken edges, flying pixels, bleeding, and aliasing.
- αDepth explicitly estimates layered foreground and background color and depth to disentangle mixed colors and resolve soft-boundary depth ambiguity.
- CAR models opacity at local soft boundaries instead of extracting globally defined foregrounds, enabling efficient scene-level inference without user intervention.
- A two-layer formulation locally separates foreground and background at soft boundaries even in complex scenes with many layers.
2 Related Work
Prior stereo-conversion and depth methods improve view synthesis but remain challenged by soft-boundary geometry. Matting methods estimate opacity yet often depend on guidance or struggle with complex, diverse scenes.
- Stereo Conversion: Stereo conversion methods use diffusion priors or spatio-temporal mechanisms to synthesize realistic views and improve temporal coherence.
- Depth Estimation: Monocular depth estimation provides dense scene geometry for stereo conversion, but conventional models generally represent each pixel with a single depth.
- Alpha Matting: Alpha matting estimates foreground opacity and can support soft-boundary modeling, while many methods rely on trimaps, masks, prompts, or other guidance.
- Challenges: Figure 3 contrasts broken edges and flying pixels from depth ambiguity with background bleeding and aliasing from single-layer refinement.
- Alpha Matting: Guidance-based matting may require user intervention or repeated passes per subject, whereas auxiliary-free methods can be category-limited and weak on diverse soft boundaries.
3 Method
αDepth combines layered color, depth, and alpha prediction with CAR-based local boundary decomposition. Its pipeline uses dual-path features, multi-branch decoding, circular alpha encoding, and soft-region-focused training.
- Problem Analysis: Soft-boundary pixels follow I = αI_FG + (1 − α)I_BG, so mixed colors and multiple targets complicate layered stereo conversion.
- Problem Analysis: Single-layer depth creates ambiguity at mixed pixels, making view transformation inaccurate and producing broken boundaries, flying pixels, bleeding, and aliasing.
- αDepth: The network uses image and depth inputs, dual semantic-detail encoders, and a multi-branch decoder for alpha, color, and depth modalities.
- Problem Analysis: Global foreground-background assignments require guidance or create sharp inter-object discontinuities, producing inaccurate alpha values known as alpha valleys.
- CAR: CAR treats opaque regions as one class and estimates opacity only at local semi-transparent boundaries rather than globally extracting foregrounds.
- CAR: CAR maps alpha using α_sin = sin(2πα) and α_cos = cos(2πα), joining α = 0 and α = 1 continuously in trigonometric space.
- CAR: During inference, predicted trigonometric components are decoded with a four-quadrant inverse tangent into alpha values α̂ ∈ [0, 1).
- Layered Representation: αDepth locally models soft boundaries with two foreground-background layers, predicting layered color and depth representations.
4 Experiments and Analysis
Experiments evaluate αDepth for stereo conversion, warping, and alpha matting. Results show strong boundary fidelity, reduced artifacts, and competitive matting performance without manual guidance.
- Experimental Settings: The experiments use Mono2Stereo and Marvel-10K for stereo conversion, plus AIM-500 and P3M-10K for circular alpha evaluation.Baselines share Depth Anything V2 on Mono2Stereo and Video Depth Anything on Marvel-10K.
- Stereo Conversion: αDepth consistently outperforms previous stereo conversion methods on soft-region and whole-image metrics, while eliminating background bleeding and achieving the best video consistency.The evaluation reports S-PSNR and S-SSIM on soft regions, alongside LPIPS, DISTS, and FVD.
- Warping: Layered modeling achieves the best warping performance compared with original VDA depth and HairGuard’s refined single-layer depth.HairGuard improves soft-boundary details but does not resolve depth ambiguity from mixed colors.
- Alpha Matting: CAR performs comparably to state-of-the-art trimap-based, mask-based, and auxiliary-free matting methods without manual guidance.Evaluation uses SAD, Grad, and Conn on unknown trimap regions after circular alpha encoding.
- Alpha Matting: CAR handles complex multi-target scenes and captures intra-object soft boundaries, extending beyond conventional single-foreground matting settings.The reported examples include multi-target scenes and intra-object soft boundaries.
5 Conclusion
The paper concludes that αDepth resolves color mixing and depth ambiguity at soft boundaries through layered representation and CAR. Its single-pass scene-level decomposition preserves geometry while recovering high-fidelity boundary structures.
- Conclusion: αDepth jointly models foreground and background colors and depths to disentangle mixed colors and resolve soft-boundary depth ambiguity.Distinct color and depth values are allocated to overlapping foreground and background layers.
- Conclusion: CAR supports local soft-boundary decomposition and automatic scene-level inference in a single forward pass without user intervention.The representation bypasses discontinuities associated with vanilla alpha in complex scenes.
- Conclusion: Layered warping projects foreground and background information separately before alpha compositing the warped view on soft-boundary regions.Depth-guided softmax splatting handles occlusions, and joint foreground projection keeps alpha aligned with foreground color.
- Conclusion: The warped image preserves geometry from state-of-the-art depth models while recovering high-fidelity structures on soft boundaries.Layered information is estimated specifically for soft-boundary regions, with zero alpha for opaque regions.
A.2 Matting Evaluation Details
The matting evaluation adapts alpha comparison to CAR’s circular representation and restricts metrics to unknown trimap regions. This enables fair comparison despite CAR’s treatment of opaque regions.
- Evaluation Protocol: Circular alpha encoding maps ground-truth and estimated alpha values into a continuous trigonometric space before metric computation.The procedure applies Eq. (2) to both labels and predictions.
- Evaluation Protocol: SAD, Grad, and Conn are each computed as the average of corresponding sine-space and cosine-space metrics.Each metric combines the two encoded channels with equal weights.
- Evaluation Protocol: The encoded metrics are evaluated only within the official trimap’s unknown regions, following ViTMatte’s protocol.SAD denotes Sum of Absolute Differences, while Grad and Conn denote gradient and connectivity losses.
- Baseline Guidance: MatAnyone 2 receives binary masks formed by thresholding ground-truth alpha maps because its required guidance is unavailable in AIM-500 and P3M-10K.ViTMatte instead uses the official trimaps.
A.3 Training Data Curation
Training data are synthesized by compositing foreground and background images and constructing layered color and depth targets with masks. The generated layers preserve foreground and background information at soft boundaries.
- Layered Color Targets: Ground-truth foreground and background colors are produced by masked blending using binary masks derived from alpha thresholds.The masks are MFG = I(α ≥ αth) and MBG = I(α ≥ 1 − αth).
- Layered Depth Targets: Depth targets are generated from original depth estimates and composed input depth, with foreground and background labels blended using the same masks.The foreground depth is defined as DFG = MFG ⊙ Dori, while background depth is generated analogously.
- Data Validation: The generated foreground and background color-depth layers preserve their respective information on soft-boundary regions.This preservation supports supervision for layered boundary decomposition.
- Training Objective: The matting loss supervises depth predictions directly and is applied independently across color channels for foreground and background outputs.Its terms include Laplacian and gradient losses.
B.1 Pixel-Level Metrics on Full Image
αDepth maintains stable soft-boundary decomposition across diverse depth inputs and camera motions, while its architectural components support accurate matting and efficient inference. Full-image metrics verify state-of-the-art performance in warping and stereo conversion.
- αDepth achieves state-of-the-art performance in both warping and stereo conversion on full-image pixel-level metrics.The evaluation follows the experimental settings of Tables 1 and 2 and reports additional metrics computed over the full image.
- αDepth preserves soft boundaries across horizontal-swing and 10 flexible camera trajectories, outperforming methods that produce broken structures or background bleeding.The comparison evaluates larger viewpoint changes and different baseline lengths.
- The model uses a multi-branch decoder to separately estimate alpha, depth, and color, reducing feature interference between modality-specific predictions.Its reported inference speed is 0.0153 seconds per 448 × 640 image on an NVIDIA GeForce RTX 4090 GPU.
- αDepth maintains stable alpha estimation and soft-boundary detail extraction across depth inputs from DAv2, DPro, PPD, and MoGe-2.The model remains consistent despite differences in the characteristics of the input depth maps.
C.2 Impact of Semantic Encoder
Semantic and geometric cues are important for resolving depth ambiguity and decomposing soft boundaries, while the model’s ablations evaluate these design choices on matting performance.
- Ablation design: The semantic encoder and edge-extraction components are evaluated through ablations on the Marvel-10K dataset.Table 10 reports the semantic-encoder and edge-extraction ablation, while Table 9 reports the multi-branch-decoder ablation.
- Camera-trajectory context: Under large viewpoint changes, αDepth achieves the best warping performance despite depth ambiguity and background bleeding in competing approaches.
- Semantic encoder: High-level contextual cues and semantic understanding are vital for resolving depth ambiguity and decomposing soft boundaries in complex scenes.
- Edge extraction: Removing depth-edge extraction can destabilize or even diverge training because depth gradients provide strong cues for soft-boundary localization.
D Limitations and Discussions
αDepth remains constrained by the quality of its initial depth map, its two-layer assumption, and the absence of explicit temporal constraints in its image-based framework.
- Dependence on Initial Depth Maps: Global scene structure and scale remain heavily dependent on the quality of the initial input depth map.Severe geometric distortions in the underlying depth model may not be fully corrected.
- Two-Layer Representation: The current two-layer model may not fully capture scenes where three or more semi-transparent layers overlap at one pixel.The paper suggests arbitrary-layer extensions and localized volumetric representations as future directions.
- Video Consistency: Because αDepth lacks explicit temporal constraints, it may produce flickering artifacts in dynamic scenes with rapidly changing depth.It may also fail to resolve boundaries at low depth gradients when targets move close together in depth.
- Societal impact: The framework automates soft-boundary decomposition for immersive 3D content creation in VR/AR, education, and entertainment applications.
F Visualization of Ablation Models
The ablations show that layered representation and CAR progressively improve soft-boundary structure preservation, while vanilla alpha representation introduces instability and artifacts.
- The baseline produces broken boundaries and flying pixels, while alpha estimation improves soft-boundary structure.
- Vanilla alpha representation suffers from alpha valley issues that degrade warping and cause unstable predictions for complex structures.
- Circular Alpha Representation circumvents alpha-valley instability and preserves image structures and soft-boundary details.
- Combining CAR with the layered representation best recovers background information at soft boundaries and achieves the strongest warping result.
- Using only L1 loss without the matting loss fails to capture fine-grained details, producing blurred and degraded soft boundaries.
G Visualization of αDepth Results
αDepth maintains soft-boundary fidelity across challenging stereo-conversion scenes and warping comparisons, including dark environments and dynamic multi-target settings. It also matches state-of-the-art matting detail without user intervention.
- αDepth performs robustly in dark environments and highly dynamic multi-target scenes.These visualizations highlight its practical value for real-world stereo-conversion applications.
- Existing methods produce broken edges, flying pixels, background bleeding, and aliasing around ambiguous soft boundaries.Single-layer depth remains a limitation even for refinement methods such as HairGuard.
- αDepth achieves superior fidelity in warping and stereo conversion by decoupling soft boundaries into local foreground and background layers.The layered representation disentangles mixed colors and resolves depth ambiguities.
- αDepth extracts intricate soft-boundary details comparably to GVM, MatAnyone 2, and ViTMatte without user intervention.The comparison covers state-of-the-art alpha-matting baselines with differing guidance requirements.