Source-linked AI summary
Decomposing Digital Paintings into Layers via RGB-space Geometry
Jianchao Tan, Jyh-Ming Lien, Yotam Gingold
TL;DR
Because final digital paintings omit their layer structure, edits such as recoloring one object can unintentionally affect overlapping regions. The paper decomposes paintings into ordered translucent paint layers using RGB-space geometry and constrained opacity optimization, producing visually faithful decompositions that support layer-based edits.
Problem
Final digital paintings do not explicitly preserve layers, making localized edits such as recoloring a coat without affecting an overlaid scarf difficult.
Method
The method extracts a simplified paint-color palette from the RGB-space convex hull, then optimizes per-pixel opacity for a user-specified layer order with translucency and spatial-coherence regularization.
Results
The decomposed layers reproduce input paintings without visually perceptible differences and enable useful layer-based edits across many examples.
Takeaways & Limitations
RGB-space geometry and constrained optimization provide useful editable layer representations for digital paintings.
Takeaways & Limitations
Opacity selection is underconstrained, and the regularized optimization may still produce layers that do not match user expectations.
Abstract
from arXiv · showhide
In digital painting software, layers organize paintings. However, layers are not explicitly represented, transmitted, or published with the final digital painting. We propose a technique to decompose a digital painting into layers. In our decomposition, each layer represents a coat of paint of a single paint color applied with varying opacity throughout the image. Our decomposition is based on the painting's RGB-space geometry. In RGB-space, a geometric structure is revealed due to the linear nature of the standard Porter-Duff "over" pixel compositing operation. The vertices of the convex hull of pixels in RGB-space suggest paint colors. Users choose the degree of simplification to perform on the convex hull, as well as a layer order for the colors. We solve a constrained optimization problem to find maximally translucent, spatially coherent opacity for each layer, such that the composition of the layers reproduces the original image. We demonstrate the utility of the resulting decompositions for re-editing.
1 Introduction
The paper addresses the difficulty of editing flattened digital paintings by recovering translucent paint layers from a single final image. It combines RGB-space geometry with constrained opacity optimization and user-specified layer ordering to support re-editing.
- Contribution: The method decomposes a flattened digital painting into translucent layers, each representing one paint color with spatially varying opacity.The layers reproduce the original painting when composited.
- Geometric basis: RGB-space geometry reveals paint-color structure because Porter-Duff “over” compositing is linear.Painted pixels move toward paint colors by linear interpolation, with transparency controlling the interpolation strength.
- Color extraction: The RGB-space convex hull provides a small palette of paint colors capable of reproducing the image’s other colors.Users can choose the degree of hull simplification.
- Opacity estimation: An optimization-based approach computes per-layer, per-pixel opacity values while balancing translucency and spatial coherence.The compositing order must be supplied because the “over” operation is not commutative.
- Application: The resulting decomposition enables structured re-editing of digital paintings that lack their original layer representation.Recovering layers avoids edits that unintentionally affect overlying or neighboring painted regions.
2 Related Work
Prior work decomposes images using editable vectors, physical or photographic assumptions, specialized brush-stroke models, or temporal editing records. This paper instead targets raster digital paintings from a single final image under Porter-Duff “over” compositing.
- Single-Image Decomposition: Unlike vector-graphics decomposition, this approach produces editable layered bitmaps for a less constrained domain with less user input.Richardt et al. [2014] require the user to select an image region for gradient-based decomposition.
- Layered Brush Strokes: Compared with Chinese-painting stroke recovery, the method handles unknown stroke shapes, more than two overlapping strokes, and varying transparency.Xu et al. [2006] assume a particular artwork style, at most two overlapping strokes, and minimally varying transparency.
- Layering Models: The method solves for opacity in the traditional globally and discretely ordered layer model rather than allowing independent orders or partial overlap.McCann and Pollard [2009; 2012] introduced those more general layering models.
- Alternative Decompositions: Its color-composition decomposition is orthogonal to scale-space and intrinsic-image methods, which separate spatial detail or illumination from reflectance.Intrinsic-image decomposition is designed for photographs of illuminated objects, not digital paintings.
- Palette Extraction: Unlike clustering-based palette extraction, the method can recover simpler hidden colors that never appear as opaque pixels in the final image.This is possible when observed colors arise from blends of original colors applied with varying translucency.
- Editing History: Unlike time-lapse approaches, it uses only the final painting and assumes that only Porter-Duff [1984] “over” operations were performed.It offers a plausible rather than necessarily true editing history.
- Matting and Reflections: Unlike matting and reflection methods, the approach does not rely on physical scene assumptions, paired photographs, or at most two layers.Digital paintings commonly contain more than two layers and do not obey photographic physical assumptions.
3 Overview
The pipeline first infers paint colors from a simplified RGB-space convex hull, then estimates opacity for those colors in a user-specified order. Energy minimization regularizes the underconstrained reconstruction toward translucent and spatially coherent layers.
- Color Identification: The color-identification stage computes the exact RGB-space convex hull and simplifies it through plane fitting followed by vertex clustering.The simplified hull vertices serve as the inferred paint colors.
- Opacity Computation: The opacity stage takes the inferred RGB colors and a user-provided order, then computes per-pixel opacity values for RGBA paint layers.The composited layers are constrained to reproduce the input image.
- Regularization: Energy minimization resolves the underconstrained polynomial reconstruction by favoring maximally translucent and spatially coherent opacity patterns.These regularizers bias the solution toward useful layer representations.
4 Identifying Paint Colors
Paint colors are inferred from the convex hull of observed pixels because linear Porter-Duff compositing keeps pixels inside the hull of original colors. Hull simplification and layer ordering remain user-guided.
- Convex-Hull Basis: Every observed pixel lies in the RGB-space convex hull of the original paint colors under Porter-Duff “over” compositing.Each pixel can be expressed as a convex combination of the original colors, although the weights are not opacity values.
- Hull Simplification: The method simplifies the observed pixel hull by fitting planes to sampled hull points, clustering resulting vertices, and treating the final vertices as paint colors.RANSAC identifies planes, while termination and inside-fraction parameters control the approximation.
- User-Guided Simplification: User involvement is required because semi-transparent colors can create overly complex hulls and no single simplification setting works well for every image.Users may adjust RANSAC parameters, mean-shift bandwidth, or remove colors directly.
- Layer Ordering: The layer order is also user-selected because “over” compositing is noncommutative and automatic ordering criteria did not match human preference.With n colors, the method faces n! possible orderings.
5 Determining Layer Opacity
The method computes per-pixel opacities for globally ordered paint colors so composited layers reproduce the observed image, while regularization selects translucent, spatially coherent solutions.
- 5 Determining Layer Opacity: At each pixel, the observed color is expressed by compositing globally ordered layer colors over an opaque background.The layer colors are fixed globally, while opacity varies per pixel.
- 5 Determining Layer Opacity: Because the opacity system can have infinitely many solutions, regularization biases the result toward translucent layers and spatial continuity.The underconstrained case arises when an opaque non-bottom layer exists or more than three layers are used.
- 5 Determining Layer Opacity: The opacity penalty favors sparse solutions by penalizing opacity while avoiding the behavior of naively minimizing squared opacity.The formulation reverses the preference near transparent and nearly opaque layers, producing sparser solutions.
- 5 Determining Layer Opacity: The spatial regularizer is Laplacian energy over opacity values, encouraging neighboring pixels’ layer opacities to vary coherently.The Laplacian measures the difference between a pixel’s opacity and the average opacity of its neighbors.
- 5 Determining Layer Opacity: The method minimizes the combined energy Epolynomial + wopaqueEopaque + wspatialEspatial subject to the polynomial constraints (Equation 2) and αi ∈[0, 1].Epolynomial is implemented as a least-squares penalty term.
- 5 Determining Layer Opacity: The recovered layers and their reconstruction are shown for the example from Figures 2 and 3.Figure 5 presents the decomposed layers and their composition.
6 Results
The decompositions reproduce varied digital paintings without visually perceptible differences and support useful layer-based edits, while optimization remains the main computational bottleneck.
- The decomposed layers reproduce varied input paintings without visually perceptible differences.Approximate convex hulls cover nearly every pixel, while polynomial constraints ensure satisfying opacity values.
- Layer representations make edits such as isolating and recoloring parts of a painting straightforward.
- A progressive multiresolution optimization reduces running time by a factor of four.The method optimizes recursively downsampled images and uses upsampled solutions as initial guesses for larger images.
- A 100 × 64-pixel image previews in less than 20 seconds, whereas 500- or 1000-pixel-wide images can require 20 minutes to an hour or longer.
- Paint-color identification takes a few seconds, but user selection of convex-hull simplification is the bottleneck.
7 Conclusion
The paper concludes that RGB-space geometry and constrained optimization yield useful translucent layer decompositions for re-editing, but the method depends on global ordering and user-selected parameters.
- RGB-space convex-hull geometry identifies paint colors, and constrained optimization decomposes paintings into useful translucent layers.
- Limitations: Opacity estimation is underconstrained, so regularization may still produce layers that do not match user expectations.
- The decompositions support structured re-editing across a variety of digital paintings.
- Limitations: The method assumes a global layer order, although a painting’s true editing history may interleave repeated applications of the same color.
- Limitations: The approach requires user choices for convex-hull simplification, layer order, and optimization weights.
- Future Work: Future work targets nonlinear edits and compositing, automatic parameter selection, physical paintings and photographs, and segmentation using layer translucency.