Source-linked AI summary
"Double-DIP": Unsupervised Image Decomposition via Coupled Deep-Image-Priors
Yossi Gandelsman, Assaf Shocher, Michal Irani
TL;DR
Many vision tasks require separating an image into component layers, but they are often treated as distinct problems. The paper introduces Double-DIP, coupling multiple Deep Image Prior networks to decompose a single image or video without training examples. It applies this framework across several tasks and reports comparable or better results than leading methods in some cases, including dehazing.
Problem
Many seemingly different computer vision tasks can be formulated as image decomposition into separate layers, motivating a unified unsupervised approach.
Method
Double-DIP couples two or more Deep Image Prior networks to reconstruct an input while producing component layers with simpler, less correlated internal statistics.
Results
The framework applies to dehazing, foreground/background segmentation, watermark removal, and transparency separation in images and videos, with comparable or better results than leading methods in some tasks.
Takeaways & Limitations
Double-DIP provides a unified unsupervised framework for varied image-decomposition tasks using only the input image or video.
Takeaways & Limitations
Transparent separation from a single image remains inherently ambiguous, especially because each pixel mixes two values and layer patch distributions are not always uniform.
Abstract
from arXiv · showhide
Many seemingly unrelated computer vision tasks can be viewed as a special case of image decomposition into separate layers. For example, image segmentation (separation into foreground and background layers); transparent layer separation (into reflection and transmission layers); Image dehazing (separation into a clear image and a haze map), and more. In this paper we propose a unified framework for unsupervised layer decomposition of a single image, based on coupled "Deep-image-Prior" (DIP) networks. It was shown [Ulyanov et al] that the structure of a single DIP generator network is sufficient to capture the low-level statistics of a single image. We show that coupling multiple such DIPs provides a powerful tool for decomposing images into their basic components, for a wide variety of applications. This capability stems from the fact that the internal statistics of a mixture of layers is more complex than the statistics of each of its individual components. We show the power of this approach for Image-Dehazing, Fg/Bg Segmentation, Watermark-Removal, Transparency Separation in images and video, and more. These capabilities are achieved in a totally unsupervised way, with no training examples other than the input image/video itself.
1. Introduction
The paper frames segmentation, transparency separation, and dehazing as image layer-decomposition problems and proposes Double-DIP, an unsupervised framework using coupled DIP networks across these tasks.
- Problem framing: Segmentation, transparency separation, and dehazing can be expressed as decomposing an image into meaningful component layers.Examples include foreground/background, reflection/transmission, and haze-free image/airlight and transmission layers.
- Internal statistics: Separate image layers have simpler, more self-similar patch distributions than the original mixed image.Prior work links internal patch recurrence and lower within-layer entropy to unsupervised decomposition tasks.
- Deep Image Prior: A single Deep Image Prior network can capture the low-level statistics of one natural image without external training examples.The DIP takes random noise as input and trains to reconstruct the image's statistics.
- Proposed framework: Double-DIP couples multiple DIPs so their outputs tend to split a mixed image into layers with simple patch distributions.The framework uses two or more DIPs, one or more per decomposed component.
- Applications: The approach is demonstrated for image dehazing, foreground/background segmentation in images and videos, watermark removal, and transparency separation in images and videos.The paper presents Double-DIP as a general-purpose framework spanning multiple image-decomposition applications.
2. Overview of the Approach
Double-DIP decomposes an image by coupling multiple DIP networks, exploiting the greater complexity of mixed-layer patch distributions and enforcing reconstruction, simplicity, and independence criteria. The framework uses reconstruction, exclusion, and task-specific regularization losses, with hints or multiple frames helping resolve ambiguities.
- Motivation: Mixed images have larger patch-distribution complexity and longer DIP reconstruction than their individual components.For the transparency example, the mixed-image loss exceeds the sum of the two individual losses; analogous behavior repeats for natural-image mixtures.
- Coupled DIPs: Multiple DIPs split image patches among outputs, with each network capturing different components of the input’s internal statistics.Shared convolutional filters promote self-similarity within each DIP output.
- Applications: The framework targets unsupervised foreground/background segmentation, transparency separation, dehazing, watermark removal, and related decomposition tasks.The segmentation formulation combines foreground and background layers with a binary pixelwise mask.
- Decomposition criteria: A meaningful decomposition recombines to the input while keeping layers simple and mutually independent.Here, simplicity means strong internal self-similarity of image elements.
- Unified architecture: Double-DIP combines reconstruction, exclusion, and task-specific regularization losses to recover layers and constrain their masks.The exclusion loss minimizes correlation between output gradients; mask regularization can enforce binary segmentation or smooth dehazing maps.
- Ambiguities and guidance: Layer separation can remain ambiguous when one layer contains multiple independent regions, often requiring crude saliency hints for initialization.For video transparency, changing mixtures across frames can resolve ambiguity because each layer’s statistics remain consistent over time.
3. Segmentation
Double-DIP frames foreground/background segmentation as unsupervised two-layer decomposition, using coupled DIPs and task-specific mask regularization for images and videos. Shared internal patch statistics across video frames encourage consistent separated layers, while the method produces high-quality segmentation without semantics or labeled training examples.
- Image segmentation: Foreground/background segmentation decomposes image I into layers y1 and y2 combined by a binary mask m(x).The formulation assumes each recovered layer is simpler to generate than the input image.
- Image segmentation: The Zebra example shows DIP-generated layers that support a good segmentation mask while filling in missing image regions.Each layer complies with the criterion that its image elements can be composed from its own pieces.
- Image segmentation: A dedicated regularization loss encourages the segmentation mask m(x) to become binary.The broader objective also combines reconstruction, gradient exclusion, and task-specific regularization.
- Image segmentation: Double-DIP obtains high-quality unsupervised segmentation but does not capture semantics, unlike supervised semantic-segmentation approaches trained on labeled examples.The comparison concerns supervision and semantic information, not only segmentation quality.
- Video segmentation: Video foreground/background segmentation uses shared internal patch statistics across sequential frames and casts the video as two-layer separation.A single DIP is shared across frames of each separated layer, while mask-DIP noise changes gradually between successive frames.
4. Transparent Layers Separation
Double-DIP separates transparent layers by modeling mixtures with separate DIP-generated layers and task-dependent masks. Single-image ambiguity can require hints, while multiple mixtures or video frames can resolve shared-layer ambiguity.
- Transparent-layer model: Transparent images are modeled as convex combinations of transmission and reflection layers using a mask m(x), often assumed uniform.Double-DIP uses separate DIPs for the layers, a third DIP for the constant mask, and an Exclusion loss to reduce layer correlation.
- Ambiguity: Single-image transparency separation has greater inherent layer ambiguity than binary segmentation because every pixel mixes two values.Successful separation is demonstrated when each layer has a relatively uniform patch distribution.
- Ambiguity: Two different mixtures of the same layers resolve ambiguity through coupled equations sharing the layer outputs while varying the mixture coefficients.One Double-DIP generates the shared layers from both mixtures simultaneously, while another DIP generates the differing coefficients.
- Video transparency: For video transparency, a static reflection and dynamic transmission are represented with one DIP per layer and another DIP for the time-varying mixing mask.The static reflection has no frame index, while its intensity may vary through m(i).
- Watermark removal: Double-DIP removes watermarks from one image with a crude bounding-box hint or from a few shared-watermark images without a bounding box.With one image, the learned mask is constrained to zero outside the supplied box; with multiple images, the watermark and mask are shared across the inputs.
5. Image Dehazing
Double-DIP treats dehazing as layer separation, recovering a haze-free image, airlight map, and transmission map from a hazy image. Its varying-airlight formulation addresses cases where uniform-airlight assumptions break and yields higher-quality dehazing.
- Dehazing model: The standard hazy-image model combines the haze-free image J(x), airlight map A(x), and transmission t(x) as I(x) = t(x)J(x) + (1 −t(x))A(x).Transmission exponentially decays with scene depth, and dehazing seeks to recover J(x).
- Double-DIP formulation: Double-DIP maps dehazing to layer separation by treating J(x) as one layer, A(x) as another, and t(x) as the mixing mask.This formulation allows the framework to estimate the image, airlight, and transmission components jointly.
- Non-uniform airlight: Uniform airlight assumptions can fail at dawn or dusk, producing distorted colors; Double-DIP instead estimates spatially varying airlight and transmission maps.The paper reports higher-quality dehazing from this varying-airlight formulation.
- Regularization: Double-DIP regularizes transmission for smoothness and constrains the airlight map toward an initial global airlight color.The transmission smoothness uses the Laplacian norm, while the airlight penalty limits deviations from the initial estimate.
- Evaluation: On O-HAZE, Double-DIP ranked second among the evaluated dehazing methods numerically, while the authors report stronger visual performance outside the dataset.The paper also reports that some NTIRE’2018 winners performed poorly outside their small challenge test set.
6. CONCLUSION
The paper presents Double-DIP as an unsupervised, general-purpose framework for image-layer decomposition across diverse tasks. It requires only the input image or video and can match or exceed leading methods on some tasks.
- Conclusion: Double-DIP handles diverse layer-decomposition tasks without training examples beyond the input image or video.The conclusion identifies dehazing, segmentation, watermark removal, and transparency separation as applications.
- Conclusion: Although general-purpose, Double-DIP achieves results comparable to or better than leading methods in some tasks, including dehazing.The authors propose adding semantic or perceptual cues as future work for higher-level tasks.