PaperlayerView exact revision on arXiv

Source-linked AI summary

STAR: A Structure and Texture Aware Retinex Model

Jun Xu, Yingkun Hou, Dongwei Ren, Li Liu, Fan Zhu, Mengyang Yu, Haoqian Wang, Ling Shao

arXiv:1906.06690v5cs.CV

TL;DR

Retinex decomposition must separate illumination from reflectance despite ambiguous image derivatives and the ill-posedness of the problem. STAR exponentiates local derivatives to form structure and texture maps that regularize the two components, using alternating optimization with closed-form least-squares subproblems. Experiments report better performance than competing methods across decomposition, low-light enhancement, and color correction.

  • Problem

    Existing Retinex methods either analyze gradients locally without global structural consistency or use global regularization while neglecting derivative properties.

  • Method

    STAR exponentiates local derivatives to extract structure and texture maps, uses them to regularize illumination and reflectance, and alternates closed-form least-squares updates.

  • Results

    STAR achieves better performance than previous competing methods on Retinex decomposition, low-light image enhancement, and color correction.

  • Takeaways & Limitations

    Exponentiated structure and texture awareness provides a global regularization framework for single-image illumination and reflectance decomposition and related applications.

Abstract

from arXiv · show

Retinex theory is developed mainly to decompose an image into the illumination and reflectance components by analyzing local image derivatives. In this theory, larger derivatives are attributed to the changes in reflectance, while smaller derivatives are emerged in the smooth illumination. In this paper, we utilize exponentiated local derivatives (with an exponent γ) of an observed image to generate its structure map and texture map. The structure map is produced by been amplified with γ > 1, while the texture map is generated by been shrank with γ < 1. To this end, we design exponential filters for the local derivatives, and present their capability on extracting accurate structure and texture maps, influenced by the choices of exponents γ. The extracted structure and texture maps are employed to regularize the illumination and reflectance components in Retinex decomposition. A novel Structure and Texture Aware Retinex (STAR) model is further proposed for illumination and reflectance decomposition of a single image. We solve the STAR model by an alternating optimization algorithm. Each sub-problem is transformed into a vectorized least squares regression, with closed-form solutions. Comprehensive experiments on commonly tested datasets demonstrate that, the proposed STAR model produce better quantitative and qualitative performance than previous competing methods, on illumination and reflectance decomposition, low-light image enhancement, and color correction. The code is publicly available at https://github.com/csjunxu/STAR.

I. INTRODUCTION

The paper identifies limitations in local or globally regularized Retinex decomposition and proposes STAR, which uses exponentiated local derivatives to obtain globally consistent structure and texture maps.

  • Motivation: Retinex decomposition separates an observed image into illumination and reflectance components, with derivatives linking larger changes to reflectance and smaller changes to smooth illumination.The simplified model is O = I ⊙R.
  • Motivation: Binary gradient classification is unreliable because illumination and reflectance changes can coincide in intermediate regions.
  • Motivation: Local gradient-based methods overlook global structural consistency, while global regularization methods may fail to exploit derivative properties for separating illumination and reflectance.
  • Proposed approach: STAR exponentiates local derivatives with exponent γ to generate amplified structure maps for γ > 1 and shrunk texture maps for γ < 1.
  • Proposed approach: The extracted structure and texture maps regularize illumination and reflectance, respectively, in a globally structured STAR decomposition model.
  • Results: Experiments report better performance than competing methods on Retinex decomposition, low-light image enhancement, and color correction.

II. RELATED WORK

The related work frames Retinex decomposition among classical, variational, and intrinsic-image approaches, then introduces the simplified model and structure-texture regularization used by STAR.

  • Retinex methods: Retinex methods include classical path-based, PDE-based, and center/surround approaches, with early path methods requiring careful tuning and incurring high computational costs.
  • Retinex methods: Variational Retinex methods have separately emphasized smooth illumination or reflectance, while some approaches omit regularization of the other component.
  • Intrinsic image decomposition: Intrinsic image decomposition separates Lambertian shading and reflectance but may lose visual scene content, limiting its suitability for simultaneous Retinex estimation in low-light enhancement.
  • Structure and texture regularization: The paper positions structure and texture awareness as the regularization strategy for illumination and reflectance in its model.
  • Simplified Retinex model: The simplified Retinex model represents an observed image as O = I ⊙R, where illumination and reflectance are estimated through regularized decomposition.

B. Structure and Texture Estimator

The estimator separates scene structure from texture to support Retinex decomposition, while addressing the vulnerability of earlier structure-texture methods to textures and edge ringing.

  • Structure and Texture Requirements: Retinex decomposition requires priors because the problem is highly ill-posed: illumination should capture object structure, while reflectance should capture texture.Texture denotes small object-surface patterns with similar local statistics.
  • Limitations of Existing Methods: TV regularizers extract object structure by enforcing gradient similarity, but prior structure-texture techniques can be vulnerable to textures and produce ringing near edges.One earlier method also fails to extract structures whose appearances resemble underlying textures.
  • Existing Filters: The TV filter uses absolute input-image gradients as a guidance map for structure-texture extraction.The paper examines TV alongside mean local variance filtering.
  • Existing Filters: Mean local variance is another filter used to estimate the structure map.Both filters are studied as typical structure-texture extraction techniques.
  • Filter Behavior: On the Value channel of an HSV-transformed image, TV and MLV filtered images broadly reflect the input image’s main structure and resemble extracted edges.The MLV and TV outputs are compared visually with an edge-detection result.

C. Proposed Structure and Texture Awareness

The proposed structure-texture awareness scheme exponentiates local derivatives to emphasize either structure or texture, then uses the resulting maps and weights in STAR decomposition.

  • Motivation: Existing TV and MLV filters tend to capture structure, so exponentiating local derivatives is introduced to reveal structure or texture more clearly.The design exploits the Retinex association between larger derivatives and reflectance changes and smaller derivatives and smooth illumination.
  • Exponential Filters: The exponentiated TV and mean local variance filters provide separate filters for structure and texture extraction.These are named the ETV and EMLV filters.
  • Exponent Effects: With γ = 0.5, ETV and EMLV roughly reveal textures, whereas γ ∈ {1, 1.5, 2} tends to extract structural edges.The exponent γ determines sensitivity to image gradients.
  • STAR Weighting: The structure and texture maps are used to construct weighting matrices for illumination and reflectance decomposition.The scheme initializes I0 and R0 as O^0.5 and uses gradient-based weights with γs > 1 and γt < 1.
  • Optimization: STAR is solved iteratively by updating illumination and reflectance in alternating steps until convergence.The algorithm initializes both components, updates them through closed-form subproblems, and outputs estimated illumination and reflectance.
  • Filter Comparison: Considering local variance enables EMLV to reveal details and preserve structures better than ETV.The paper states that this difference is validated experimentally.

A. Proposed Model

STAR jointly estimates illumination and reflectance by regularizing structure and texture maps, then solves alternating convex sub-problems with closed-form least-squares updates.

  • Model formulation: STAR simultaneously estimates illumination I and reflectance R of an observed image O using structure- and texture-aware regularization.The model uses TV ℓ2-norm regularization for both components.
  • Alternating optimization: The alternating optimization updates I while fixing R, then updates R while fixing I, because the objective is separable in the two variables.Each update is treated as a convex sub-problem.
  • Closed-form updates: Each sub-problem is vectorized into a standard least-squares regression using gradient and diagonal weighting matrices.For the illumination update, vec(·), a forward-difference gradient matrix G, and diagonal matrices Drk and Ds0 are introduced; the reflectance update is reformulated similarly.
  • Stopping and convergence: The algorithm stops when the relative Frobenius-norm change in either component reaches ε or when the maximum iteration count K is reached.The experiments use ε = 10^-2 and K = 20.
  • Stopping and convergence: The convergence analysis reports that either relative update error falls below ε = 0.01 within 10 iterations on 35 low-light images.The overall objective is described as convex with a global optimal solution.

C. Updating Structure and Texture Awareness

STAR alternates between updating the structure and texture maps and solving for illumination and reflectance, yielding overall complexity O(nmKL).

  • Updating Structure and Texture Awareness: After decomposing O = I ⊙R, STAR updates structure S and texture T before resolving illumination I and reflectance R for L iterations.The experiments set L = 4 to balance speed and accuracy.
  • Complexity Analysis: The overall computational complexity for an n×m image is O(nmKL), where K and L are the iteration counts of the two update procedures.Algorithm 1 updates I and R, while Algorithm 2 updates S and T.

V. EXPERIMENTS

The experiments evaluate STAR's Retinex decomposition and use HSV processing with empirically selected regularization and exponent parameters.

  • Experimental setup: The experiments assess STAR qualitatively and quantitatively for Retinex decomposition, including an ablation study of its components.Experiments run on a Huawei Matebook X Pro with an Intel Core i5 8265U CPU and 8GB memory.
  • Experimental setup: STAR processes only the Value channel after converting RGB images to HSV, then converts the processed result back to RGB.The reported parameters are α = 0.001, β = 0.0001, γs = 1.5, and γt = 0.5.

B. Retinex Decomposition

The Retinex decomposition experiments compare STAR with established methods despite difficult ground-truth evaluation, examining quality, speed, weighting choices, and update iterations.

  • B. Retinex Decomposition: Accurate Retinex decomposition requires smooth illumination that preserves structure and reflectance that reveals scene details, but component ground truths are difficult to generate.This makes quantitative comparison of existing methods difficult.
  • B. Retinex Decomposition: STAR is qualitatively compared with the Baseline, MSR, SIRE, WVM, JieP, RRM, and RDGAN for illumination and reflectance estimation.The decomposition is performed on the HSV Value channel, similarly to the compared methods.
  • Comparison on speed: STAR is faster than WVM and RRM but slower than JieP and SIRE on a 960 × 720 RGB image.The timing comparison averages 10 runs for each method.
  • Comparison on speed: Although not the fastest method, STAR achieves better decomposition performance than methods such as JieP and SIRE.The reported comparison is qualitative at this passage's level of detail.
  • Ablation study: STAR-EMLV maintains structure and texture better than STAR-ETV, whose results tend to contain little illumination structure and lose reflectance texture.The ablation compares two weighting schemes within STAR.
  • Ablation study: With more structure-texture updating iterations, illumination becomes more structural while reflectance presents more details.The study evaluates L = 1, 2, and 4.

A. Low-light Image Enhancement

STAR is evaluated for low-light enhancement using objective metrics and visual comparisons, including ablations of weighting, structure/texture awareness, parameters, and update iterations. It achieves better reported visual-quality metrics than competing methods, with clear illumination enhancement and comparable noise suppression.

  • Low-light enhancement targets improved visibility and contrast while preserving color information through Retinex processing in the HSV Value channel.
  • The experiments examine weighting schemes, structure and texture awareness, γ parameters, α and β selection, and iterative map updates.
  • STAR achieves lower NIQE and higher VIF than competing methods on the evaluated low-light datasets.Lower NIQE and higher VIF indicate better reported visual quality.
  • STAR produces visually clear content while naturally enhancing illumination and achieves comparable noise suppression with competing methods on representative images.

VII. CONCLUSION

The paper concludes that STAR combines exponential structure/texture extraction with alternating Retinex estimation and outperforms representative methods across several tasks. Direct benchmarking of decomposed illumination and reflectance remains limited because suitable ground truths are unavailable.

  • STAR uses an EMLV filter to extract structure and texture maps, regularizes illumination and reflectance, and alternates map updates with component estimation.
  • Comprehensive experiments report better quantitative and qualitative performance than representative Retinex decomposition methods on decomposition, low-light enhancement, and color correction.
  • Direct Retinex benchmarking is constrained by the lack of reasonable ground truths for decomposed illumination and reflectance.The authors identify synthetic datasets with ground truths as future work.
Loading 1906.06690v5…