PaperlayerView exact revision on arXiv

Source-linked AI summary

Guardians of the Hair: Rescuing Soft Boundaries in Depth, Stereo, and Novel Views

Xiang Zhang, Yang Zhang, Lukas Mehl, Markus Gross, Christopher Schroers

arXiv:2601.03362v1cs.CV

TL;DR

Soft-boundary pixels mix foreground and background, making 3D estimation ambiguous and difficult. HairGuard combines matting-informed depth refinement with a view-synthesis pipeline, and reports state-of-the-art performance across depth, stereo conversion, and novel-view synthesis.

  • Problem

    Mixed foreground and background pixels at soft boundaries create uncertain correspondence and depth ambiguity, challenging pixel-wise 3D estimation.

    Source: §1 · p. 2

  • Method

    HairGuard uses matting datasets to train a gated-residual depth fixer and combines fixed-depth warping, scene-painter inpainting, and color fusion for view synthesis.

    Source: §1 · p. 2, §3 · p. 4, §3.1 · p. 5

  • Results

    HairGuard is reported to achieve state-of-the-art performance across monocular depth estimation, stereo image/video conversion, and novel-view synthesis.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The gated depth fixer is designed for plug-and-play integration with state-of-the-art depth models while preserving soft-boundary detail.

    Source: §3.1 · p. 5

    The discussion identifies occlusions and overlapping surfaces as a remaining challenge and suggests dense 3D representations such as 3D Gaussians.

    Source: E. Limitation and Discussion · p. 14

Abstract

from arXiv · show

Soft boundaries, like thin hairs, are commonly observed in natural and computer-generated imagery, but they remain challenging for 3D vision due to the ambiguous mixing of foreground and background cues. This paper introduces Guardians of the Hair (HairGuard), a framework designed to recover fine-grained soft boundary details in 3D vision tasks. Specifically, we first propose a novel data curation pipeline that leverages image matting datasets for training and design a depth fixer network to automatically identify soft boundary regions. With a gated residual module, the depth fixer refines depth precisely around soft boundaries while maintaining global depth quality, allowing plug-and-play integration with state-of-the-art depth models. For view synthesis, we perform depth-based forward warping to retain high-fidelity textures, followed by a generative scene painter that fills disoccluded regions and eliminates redundant background artifacts within soft boundaries. Finally, a color fuser adaptively combines warped and inpainted results to produce novel views with consistent geometry and fine-grained details. Extensive experiments demonstrate that HairGuard achieves state-of-the-art performance across monocular depth estimation, stereo image/video conversion, and novel view synthesis, with significant improvements in soft boundary regions.

1. Introduction

HairGuard targets the difficulty of reconstructing soft boundaries in 3D vision, where mixed foreground and background pixels make depth and correspondence ambiguous. It combines matting-informed depth refinement with view-synthesis components to recover fine-grained details.

1. Introduction

  • Soft-boundary pixels mix foreground and background, creating uncertain correspondence and depth ambiguity that make pixel-wise 3D estimation ill-posed.Such boundaries occur with thin or semi-transparent structures and alpha blending in rendered imagery.Source: §1 · p. 2
  • Depth-based methods miss fine hair detail or detach hairs, while depth-free generative methods risk hallucinated or degraded soft-boundary textures.The introduction attributes texture degradation in some generative methods to pixel-to-latent compression.Source: §1 · p. 2
  • HairGuard uses matting datasets to train a depth fixer that automatically identifies soft boundaries and refines depth with a gated residual module.The design supports plug-and-play integration with zero-shot depth models and avoids relying on manually crafted cues such as trimaps.Source: §1 · pp. 2–3
  • For view synthesis, HairGuard forward-warps fixed depth, uses a scene painter for disocclusions, and applies a dual-skip color fuser to preserve detail and geometric consistency.The color fuser is intended to address texture hallucination and detail compression in generative models.Source: §1 · pp. 2–3
  • Experiments report effectiveness across monocular depth estimation, stereo image/video conversion, and novel-view synthesis.The paper describes HairGuard as superior across these 3D vision tasks.Source: §1 · p. 2

2. Related Work

Related work traces persistent soft-boundary challenges across depth estimation, stereo conversion, and novel-view synthesis. HairGuard combines depth correction with texture-restoring view-synthesis components to address these gaps.

2.1. Monocular Depth Estimation

  • Despite detail-focused advances such as Depth Anything V2, Depth Pro, and UniDepthV2, depth methods still produce missing or discontinuous estimates around soft boundaries.HairGuard is presented as localizing these boundaries and reconstructing fine-grained depth details.Source: §2.1 · p. 3

2.2. Stereo Conversion

  • Stereo conversion methods can hallucinate textures or lose fine details; HairGuard’s color fuser restores detail after fixed-depth warping and scene-painter inpainting.The pipeline uses the depth fixer before warping, then combines warped and inpainted results for novel views.Source: §2.2 · p. 3, Figure 3 · p. 3

2.3. Novel View Synthesis

  • Diffusion-based novel-view synthesis can suffer texture inconsistency, while depth-guided SplatDiff depends on depth quality that is often poor around soft boundaries.HairGuard pairs a depth fixer with a color fuser to correct depth inaccuracies and restore fine-grained texture details.Source: §2.3 · p. 3

3. HairGuard

HairGuard uses matting data to model soft boundaries, with a depth fixer refining depth and a view-synthesis pipeline combining warped and inpainted views. Its depth training targets boundary detail while preserving global depth quality, and its color fuser addresses artifacts in both input views.

3.1. Depth Estimation

  • Matting alpha mattes and background images yield synthetic depth pairs, with threshold variation creating fine-boundary ground truth and broken or missing boundary inputs.The depth fixer is trained to identify and correct soft boundaries without relying on hand-crafted cues such as trimaps.Source: §3.1 · pp. 4–5
  • The depth fixer uses image semantics, depth gradients, and a predicted gate to localize soft boundaries and apply residual corrections there, supporting plug-and-play refinement.The gated residual performs better than direct prediction or a vanilla residual when combined with two-stage training.Source: Figure 4 · p. 4, §3.1 · p. 5
  • Two-stage training first emphasizes masked soft-boundary detail, then applies the alpha loss globally to preserve overall depth quality.The first-stage loss can introduce halo artifacts; the global constraint is applied in the second stage.Source: §3.1 · p. 5

3.2. View Synthesis

  • For view synthesis, HairGuard forward-warps fixed-depth views, uses a scene painter to fill disocclusions, and fuses warped and inpainted views to address background leakage and hallucinated textures.The color fuser uses a dual-skip module built on a pretrained VAE to combine the inputs.Source: §3.2 · pp. 5–6, Figure 5 · p. 5

4. Experiments and Analysis

Experiments evaluate HairGuard on depth estimation, stereo conversion, and novel view synthesis. It improves soft-boundary depth and stereo results while preserving zero-shot depth performance, and user-study results favor its novel views.

4.1. Experimental Settings

  • Evaluation covers five zero-shot depth benchmarks, Marvel-10K stereo image and video conversion, and two natural-image matting datasets for novel view synthesis.Marvel-10K contains 12,525 stereo pairs; the depth fixer was trained with matting and multi-view datasets, and the scene painter and color fuser were fine-tuned separately.Source: §4.1 · p. 6, §4.3 · p. 7, §4.4 · p. 8

4.2. Depth Estimation

  • Depth fixer improves soft-boundary depth accuracy across three depth models while retaining comparable or slightly better zero-shot performance on five unseen datasets.The gated residual module adaptively refines soft boundaries while maintaining the base model’s zero-shot performance, enabling plug-and-play integration.Source: §4.2 · pp. 6–7, Figure 6 · p. 7, Table 2 · p. 7

4.3. Stereo Conversion

  • HairGuard consistently outperforms existing approaches across all reported stereo conversion metrics, despite targeting soft boundaries that occupy small image regions.The ablation reports that depth fixing improves SIoU, scene painting improves LPIPS but worsens PSNR, and color fusion combines warped and inpainted results for the best results.Source: §4.3 · p. 8, Table 3 · p. 8, Table 4 · p. 8

4.4. Novel View Synthesis

  • In a user study with 27 participants and 1,332 votes, the results favor HairGuard over previous state-of-the-art approaches for novel view synthesis.The study used the full AIM-500 and P3M-10K evaluation sets, totaling 1,000 natural images, without hand-picked samples.Source: §4.4 · p. 8, Figure 8 · p. 8, Table 5 · p. 8

5. Conclusion

HairGuard addresses soft-boundary challenges across 3D vision tasks with a depth fixer, scene painter, and color fuser. Experiments report state-of-the-art performance in depth estimation, stereo conversion, and novel view synthesis.

5. Conclusion

  • HairGuard uses matting datasets to train a depth fixer that identifies and corrects soft-boundary depth, while scene painting and color fusion address geometry and texture in novel views.Experiments report state-of-the-art performance across monocular depth estimation, stereo image/video conversion, and novel view synthesis.Source: §5 · p. 8

Supplementary Material

The supplementary material covers implementation details, further performance analyses, additional experiments, limitations, future directions, and visual comparisons.

Supplementary Material

  • The supplement organizes material into implementation details, robustness, plug-and-play performance, computational complexity, further depth-fixer and color-fuser experiments, limitations, future directions, and visual comparisons.Visual comparisons cover ablations, monocular depth estimation, stereo conversion, and novel view synthesis.Source: Supplementary Material · p. 9

A.1. Marvel-10K Dataset

Marvel-10K provides challenging cinematic stereo sequences. The supplement also details color-fuser feature fusion and reports depth-fixer robustness on unseen scenes.

A.1. Marvel-10K Dataset

  • Marvel-10K contains 501 stereo sequences from five Marvel films, with 25 stereo pairs per shot and challenging cinematic content.The dataset includes computer-generated characters, intense motion, complex lighting, and uncommon scenes; left views serve as input and right views as ground truth.Source: §A.1 · p. 9, Figure 9 · p. 9

A.2. Color Fuser

  • The color fuser combines multi-scale warped and inpainted features with warped masks through residual decoder blocks.Its fused features are injected into the VAE decoder in a residual fashion, and the additional residual blocks use zero initialization during training.Source: §A.2 · p. 9

B.1. Robustness and Generalization

  • On unseen Marvel-10K scenes, the depth fixer maintains base-model depth quality where soft boundaries are absent and shows promising boundary refinement in complex scenes.The authors attribute generalization to decoupling depth estimation from soft-boundary refinement and using a gated residual mechanism.Source: §B.1 · p. 9

B.2. Plug-and-Play Performance

The depth fixer transfers across image- and video-based depth models, while HairGuard achieves state-of-the-art novel-view-synthesis performance with competitive computational efficiency. The section also reports stable soft-boundary refinement and identifies the scene painter as the framework’s main computational cost.

B.2. Plug-and-Play Performance

  • The depth fixer improves predictions from Depth Pro and UniDepthV2 despite their different boundary-error patterns.It is also applied to video depth models trained without additional depth-fixer retraining, preserving temporal consistency and showing stable boundary detection in occluded scenes.Source: §B.2 · pp. 9–10, Figure 11 · p. 11
  • Combining the depth fixer with SplatDiff yields consistent performance gains across all reported Marvel-10K metrics.The cited comparison is Table 6.Source: §B.2 · p. 10, Table 6 · p. 10

B.3. Computational Complexity

  • The scene painter is the largest computational-cost contributor, and its replacement with a lighter variant is proposed as an efficiency option.HairGuard’s three components run sequentially, so peak GPU memory matches the scene painter’s requirement.Source: §B.3 · p. 11
  • HairGuard achieves state-of-the-art performance while maintaining competitive computational efficiency on the Marvel-10K comparison.The scene painter dominates computational cost, and because components run sequentially, it determines peak GPU memory.Source: §B.3 · pp. 11–12, Table 7 · p. 12

C. More Experiments on Depth Fixer

These experiments evaluate how depth-fixer design choices affect soft-boundary warping and how color-fuser choices affect view synthesis. They report gains from gated residual refinement, matting loss, pretrained model priors, edge guidance, a lower alpha threshold, a VAE prior, and dual skips.

C.1. Warping Performance

  • The depth fixer consistently improves soft-boundary warping across three base depth models on Marvel-10K.Metrics are computed only within predicted soft-boundary regions; the cited comparison is Table 8.Source: §C.1 · p. 12

C.2. Gated Residual

  • Gated residual refinement localizes soft boundaries and outperforms direct prediction and vanilla residual mechanisms.The alternatives can cover redundant background and miss fine detail; the comparison is Table 9, experiments #3 versus #1–2.Source: §C.2 · p. 12, Table 9 · p. 12

C.3. Loss Function

  • Combining the ℓ1 and image matting losses performs better than using ℓ1 loss alone.Other training settings are held unchanged in the comparison of Table 9, experiments #5 versus #4.Source: §C.3 · p. 12, Table 9 · p. 12

C.4. Model Prior

  • Initializing the feature branch with pretrained Depth Anything V2 yields strong performance with about 20K training samples.The reported comparison is Table 9, experiments #7 versus #6.Source: Table 9 · p. 12, §C.4 · p. 13

C.5. Edge Guidance

  • Depth-gradient edge guidance improves soft-boundary localization and warping performance.The comparison is Table 9, experiments #9 versus #8.Source: Table 9 · p. 12, §C.5 · p. 13

C.6. Alpha Threshold

  • A lower alpha threshold preserves low-opacity details such as thin hair and gives the best reported warping performance.A higher threshold reduces redundant background but can omit faint details; it may suit tasks preferring precise depth boundaries, such as 3D segmentation or point-cloud reconstruction.Source: Table 9 · p. 12, §C.6 · p. 13, Figure 12 · p. 13

D.1. VAE Prior

  • The color fuser with a pretrained VAE prior significantly outperforms its from-scratch counterpart in visual quality.The comparison is reported in Table 10, experiments #2 versus #1.Source: §D.1 · p. 13, Table 10 · p. 13

D.2. Dual Skip

  • The dual skip module achieves the best reconstruction performance by using fine-grained features from both warped and inpainted images.A single skip improves over no skips by alleviating detail compression in VAE encoding; the comparison is in Table 10, experiments #3–5.Source: §D.2 · p. 13, Table 10 · p. 13

E. Limitation and Discussion

HairGuard’s depth fixer is limited to errors within soft-boundary regions, while its single-layer depth representation can leave geometric distortions in scenes with multiple depth layers per pixel. The discussion notes that layered outputs showed limited generalization and identifies dense 3D representations as a possible direction for handling occlusions and overlapping surfaces.

E. Limitation and Discussion

  • The gated depth fixer cannot correct depth errors outside the soft-boundary regions it locates.The authors suggest training a model-specific fixer as a possible way to adapt better to the base model.Source: E. Limitation and Discussion · p. 14
  • Single-layer forward warping may distort geometry when a pixel contains multiple depth layers, as with transparent objects.Layered foreground/background outputs with opacity helped in some cases but had limited generalization, likely due to increased estimation complexity.Source: E. Limitation and Discussion · p. 14
  • Dense 3D representations, such as 3D Gaussians, are proposed as a possible direction for handling occlusions and overlapping surfaces.The authors also identify larger training datasets as a possible route to a stronger prior for robust layered-output performance.Source: E. Limitation and Discussion · p. 14

F.1. Ablation Study

The qualitative results describe HairGuard’s component roles and report fine-grained depth recovery, high-quality stereo conversion, and state-of-the-art novel-view synthesis. The color fuser combines warped and inpainted content to maintain texture and geometry when the inpainted results contain inconsistent details.

F.1. Ablation Study

  • The ablation illustrates the roles of depth fixing, scene painting, and color fusion in producing consistent geometry and high-fidelity texture.Depth correction preserves hair structures in warped images; painter outputs can differ in texture because of diffusion hallucination and pixel-to-latent compression.Source: §F.1 · p. 14, Figure 14 · p. 15

F.2. Monocular Depth Estimation

  • The depth fixer recovers sharp, fine-grained depth details even in challenging cases with very thin hair.The qualitative results cover AIM-500 and P3M-10K.Source: §F.2 · pp. 14–15

F.3. Stereo Conversion

  • HairGuard produces stereo conversions with consistent geometry and texture, retaining details that prior generative approaches may hallucinate or degrade.The visual comparison is on Marvel-10K.Source: §F.3 · p. 15

F.4. Novel View Synthesis

  • HairGuard achieves state-of-the-art novel-view synthesis by fixing soft-boundary depth and recovering high-fidelity texture with the color fuser.The cited comparisons include ViewCrafter, ReCamMaster, and SplatDiff on AIM-500 and P3M-10K.Source: §F.4 · p. 15
Loading 2601.03362v1…