Iterative Gaussianization: from ICA to Random Rotations

I. INTRODUCTION

• High-dimensional PDF estimation is difficult because direct sampling is impeded by the curse of dimensionality, prompting problem-oriented PDF models.Many signal-processing tasks depend on an appropriate PDF description.Source: §I · p. 1
• PCA and linear ICA rely on restrictive distribution assumptions, and residual dependencies can remain after their transforms.PCA assumes a Gaussian source, while linear ICA assumes independent non-Gaussian sources.Source: §I · p. 1
• A differentiable transform to a known PDF can support estimation of the unknown PDF in the original domain, even if the transform is qualitatively meaningless.An infinite family of transforms may be suitable, allowing selection of the most convenient one.Source: §I · p. 1, §II · p. 2
• The proposed method targets a unit-covariance Gaussian with iterative transforms based on arbitrary rotations, using negentropy properties to guarantee a Gaussianization transform across rotation classes.The target is chosen because its spherical symmetry and rotations support a cost function with useful theoretical properties.Source: §I · p. 1

II. MOTIVATION

• Reducing marginal negentropy is a univariate task, whereas removing multi-information requires a difficult multivariate search; RBIG therefore focuses on marginal Gaussianization and uses rotation to revisit it.Projection Pursuit instead seeks interesting independent components, a computationally demanding task in the cited discussion.Source: §II · pp. 2–3

III. ROTATION-BASED ITERATIVE GAUSSIANIZATION (RBIG)

• The section introduces the basic formulation of RBIG.The section roadmap begins with the method's basic formulation.Source: §III · p. 3
• It analyzes differentiability, invertibility, and convergence.These are the properties identified for analysis in the section roadmap.Source: §III · p. 3
• It discusses the role of the rotation matrix used in the scheme.Source: §III · p. 3

A. Iterative Gaussianization based on arbitrary rotations

• Each RBIG iteration marginally Gaussianizes every dimension and then applies a generic rotation to the transformed variable.The initial variable follows an unknown PDF.Source: A. Iterative Gaussianization based on… · p. 3
• Any suitable rotation can be chosen because the method relies on marginal PDF estimates, not interpretation of the Gaussianized domain.The point-dependent Jacobian is needed when working in that domain, while the method's original-domain interpretation is unaffected.Source: A. Iterative Gaussianization based on… · p. 3

B. Invertibility and differentiation

• Invertibility permits sampling back in the original domain, and the transform's Jacobian supports estimating its PDF there; differentiability does not depend on the chosen rotations.The Jacobian across iterations is formed from the corresponding iteration Jacobians.Source: B. Invertibility and differentiation · pp. 3–4
• Marginal Gaussianization combines CDF-based uniformization with Gaussianization through the inverse CDF of a univariate Gaussian.The transforms are applied dimension-wise at each iteration.Source: B. Invertibility and differentiation · p. 3
• The implementation estimates marginal transforms from cumulative histograms without assuming a particular model, while alternative parametric models and their effects are left unexplored.Parametric models may improve regularization and avoid overfitting, but can make the model more rigid.Source: B. Invertibility and differentiation · p. 3
• Inversion is assured when each marginal PDF has connected support, so its CDF is strictly monotonic; problems arise when distant clusters have nonoverlapping projections onto the axes.Holes in the joint support alone are not a problem if the marginal supports remain connected.Source: B. Invertibility and differentiation · pp. 3–4

C. Convergence properties

• Marginal Gaussianization reduces negentropy, and a subsequent rotation does not change that reduction.The stated decrease equals the marginal negentropy and is nonnegative for every rotation.Source: C. Convergence properties · p. 4
• Any rotation reduces redundancy among a marginally Gaussianized variable’s coefficients.The reduction also applies to the combined Gaussianization-and-rotation step because dimension-wise Gaussianization preserves multi-information.Source: C. Convergence properties · p. 4
• These properties imply that Gaussian distance decreases each iteration and redundancy is reduced, supporting convergence for a wide class of rotations.The paper states that the distance to a Gaussian decays to zero for this class.Source: C. Convergence properties · p. 4

D. On the rotation matrices

• ICA gives the optimal convergence rate by maximizing marginal non-Gaussianity, but lacks a closed-form solution and can be computationally demanding.Its negentropy reduction is strictly positive until the Gaussian is reached.Source: D. On the rotation matrices · p. 5
• PCA is closed-form and fast but converges sub-optimally; in a singular case, it can remain stuck at a non-Gaussian local minimum.The exception is an input that is non-Gaussian despite having unit-covariance Gaussian marginals; the authors report that it did not occur in their real datasets.Source: D. On the rotation matrices · p. 5
• Random rotations guarantee theoretical convergence, with nonnegative but sub-optimal negentropy reduction; their per-transform cost does not depend on sample count.A rotation can serve as a safety check after PCA reaches zero marginal non-Gaussianity, and its computation is negligible relative to marginal Gaussianization.Source: D. On the rotation matrices · p. 5
• Table I compares Gaussianization properties across rotation choices, including convergence and computational cost.The table’s caption identifies its scope; the text discusses ICA, PCA, and random rotations.Source: D. On the rotation matrices · p. 4, Table I · p. 5

IV. RELATION TO OTHER METHODS

• The section considers RBIG’s relation to iterative projection-pursuit-like Gaussianization methods.These methods are cited as –.Source: §IV · p. 5
• It also compares RBIG with direct Gaussianization approaches designed for particular PDFs.The cited approaches are,, and.Source: §IV · p. 5
• The discussion includes connections to Support Vector Domain Description and deep neural networks.The cited methods are and.Source: §IV · p. 5

A. Iterative Projection Pursuit Gaussianization

• Unlike projection pursuit, RBIG applies marginal Gaussianization before the multivariate rotation, rather than first finding interesting projections.Projection pursuit removes structure in an ordered transition; RBIG follows a disordered transition to disorder,,.Source: A. Iterative Projection Pursuit… · p. 5

B. Direct (single-iteration) Gaussianization algorithms

• Direct Gaussianization can work for PDFs Gaussianizable along suitable axes or for elliptically symmetric PDFs,,.The method in seeks Gaussianizable components, while Radial Gaussianization targets elliptical distributions.Source: B. Direct (single-iteration)… · pp. 5–6
• RG reduces neighbor-pixel mutual information more than RBIG for natural photographic and visible Landsat images, while RBIG is more robust on SAR data.RG does not significantly reduce mutual information for the nonnatural radar image; RBIG makes no PDF assumption in these examples.Source: Figure 2 · p. 6, B. Direct (single-iteration)… · p. 6
• Radial Gaussianization can fail when the data are not elliptically symmetric, including particular images that do not follow that assumption.The method equalizes the histogram of data magnitudes to that of a Gaussian.Source: B. Direct (single-iteration)… · p. 6

C. Relation to Support Vector Domain Description

• RBIG estimates a probability density, whereas SVDD estimates a separation boundary; RBIG’s point-dependent Jacobian means its optimal rejection boundary need not be spherical.Source: C. Relation to Support Vector Domain… · p. 6

D. Relation to Deep Neural Networks

• RBIG’s dimension-wise nonlinear transforms and linear transforms resemble neural-network layers, and early stopping can act as model regularization.The authors identify this connection as a potential future research direction.Source: D. Relation to Deep Neural Networks · p. 6

V. EXPERIMENTAL RESULTS

• The experiments illustrate RBIG in convergence analysis, mutual-information estimation, image synthesis, classification, and denoising, with comparisons to standard methods in each application.Source: §V · p. 7

A. Method convergence and early-stopping

• RBIG converges to a multivariate Gaussian with RND, PCA, and ICA, although ICA needs fewer iterations, PCA follows closely, and RND is slower and more variable early on.The convergence plots show cumulative negentropy reduction and Gaussianity-test results across iterations.Source: Figure 4 · p. 7, A. Method convergence and early-stopping · p. 7
• ICA rotations increase convergence time, making PCA a good trade-off between Gaussianization error and computational cost when iterations are properly chosen.The paper suggests early stopping based on cumulative negentropy reduction or a multivariate Gaussianity test.Source: A. Method convergence and early-stopping · p. 7

• The 2D experiments compare RBIG with naive histogram-based and Rudy estimates, using synthetic distributions with known multi-information.Experiments use 10,000 samples and 10 realizations for each distribution.Source: B. Multi-information Estimation · p. 8
• RBIG outperforms other methods for Gaussian and exponential-Gaussian distributions, while naive estimation gives better estimates for marginally uniform data.Source: B. Multi-information Estimation · p. 8
• In multidimensional experiments, RBIG estimates are compared with actual multi-information, and absolute errors increase slightly with dimensionality.Source: B. Multi-information Estimation · p. 8

C. Data Synthesis

• RBIG learns an invertible transform from training data, then synthesizes samples by mapping zero-mean unit-covariance Gaussian samples back into the original domain.Source: C. Data Synthesis · p. 8

1) Toy data:

• Toy-data examples show that samples generated by inverting Gaussian samples approximately follow the original probability density.The figure shows non-Gaussian data, its Gaussianized form, and synthesized data.Source: Figure 5 · p. 8, 1) Toy data · p. 8

2) Face synthesis:

• RBIG-generated face samples have a similarity distribution suggesting greater realism than samples generated with RG.The comparison uses inner-product similarity between 200 actual and synthesized images, averaged over 300 realizations.Source: 2) Face synthesis · p. 8, Figure 7 · p. 9

D. One-class Classification

• RBIG substantially outperforms SVDD with moderate or large training sets; 750 samples suffice in a 10-dimensional problem when little is known about the non-target class.Source: Figure 8 · p. 9, D. One-class Classification · p. 9
• RBIG better rejects non-urban areas in the Naples 1995 maps, while neighboring-pixel noise may reflect the lack of spatial information.The authors suggest postclassification smoothing or spatial texture features as possible remedies.Source: D. One-class Classification · pp. 9–10, Figure 9 · p. 10

• RBIG estimates the natural-image probability model used in Bayesian denoising, addressing the difficulty of modeling high-dimensional image PDFs with traditional methods.The setup frames denoising as estimating an image from an additively degraded observation.Source: E. Image Denoising · p. 10
• The image prior is estimated from 100 achromatic images using wavelet subbands and patches, with posterior expectations approximated from 8,000 samples per coefficient.The expected value over posterior samples corresponds to using the L2 norm.Source: E. Image Denoising · p. 10
• RBIG obtains better numerical and visual denoising results than hard- and soft-thresholding in the Barbara image example.The image is corrupted with Gaussian noise of σ_n^2 = 100; evaluation uses RMSE and SSIM.Source: E. Image Denoising · p. 10

VI. CONCLUSIONS

• RBIG uses differentiable transforms to a Gaussian so the unknown PDF can be computed at any original-domain point using the transform Jacobian.The iterative transform applies univariate marginal Gaussianization followed by a rotation.Source: §VI · p. 10
• A wide class of orthonormal transforms, including random rotations, is suited to Gaussianization, allowing rotation choice to focus on practical convenience rather than interesting projections.The paper illustrates RBIG in image synthesis, classification, denoising, and multi-information estimation.Source: §VI · p. 10
• Fig. 10 presents original, noisy (noise variance σ2 n = 100), and restored Barbara images, with RMSE and SSIM given in parentheses.The caption names the measures but supplies no values in the passage.Source: Figure 10 · p. 11
• Choosing rotation families for a good Jacobian and criteria that ensure generalization are identified as topics for future research.The passage frames these as application-specific issues.Source: §VI · p. 11