Dale meets Langevin: A Multiplicative Denoising Diffusion Model

1 Introduction

• Observed log-normal synaptic weights and Dale’s law motivate asking for a sampling analogue of EGD, which preserves neuronal sign constraints.The introduction notes that standard gradient descent does not guarantee sign preservation, while EGD respects Dale’s law and produces log-normal weights.Source: §1 · pp. 1–2

1.1 Related Works

• Prior GBM work addressed sequential denoising rather than image generation; this paper develops reverse-time SDEs and samplers to complete the generative process.The paper contrasts its GBM-based approach with Vuong and Nguyen [2024], who use a log transform for sequential denoising.Source: §1.1 · p. 3

1.2 Main Contributions

• The framework uses GBM and derives reverse-time SDEs in ambient and log-transformed spaces, yielding sign-agnostic and sign-preserving samplers.The sign-preserving sampler, called Dale-Langevin, is derived via the Lamperti transform.Source: §1 · p. 2, §1.2 · p. 3
• M-DSM and M-ESM are equivalent, non-negative score matching is a special case at t = 0, and the EGD-driving convex function matches the one governing Dale-Langevin dynamics.Experiments validate image generation on MNIST, Fashion-MNIST, Kuzushiji-MNIST, and CIFAR-10.Source: §1 · p. 2, §1.2 · pp. 3–4

2 Stochastic Differential Equations and Generative Modelling

• Diffusion generation reverses a stochastic forward process, matching the forward and reverse marginal densities at each time rather than merely reversing time.The forward process maps samples from the unknown data density to an easy-to-sample noise density.Source: §2 · p. 4
• Unlike standard formulations with an isotropic Gaussian terminal density, this framework uses a log-normal marginal density.The passage states that the diffusion framework permits any forward SDE with a tractable terminal density.Source: §2 · p. 4
• Generating from the reverse-time SDE requires the time-dependent score, approximated by a neural network trained with denoising score matching.The conditional score target is determined by the forward SDE, and the loss includes a time-dependent stabilizing weight.Source: §2 · p. 5

3 Geometric Brownian Motion and its Time-reversal as a Generative Model

• GBM has log-normal marginals and preserves positivity, making it a natural forward process for positive-valued data.Its relative increments follow Brownian motion, unlike additive processes with Gaussian marginals.Source: §3 · p. 5
• The element-wise GBM forward process yields log-normal entries and a closed-form terminal distribution from which samples can be drawn.For image data, the paper uses an element-wise multivariate extension.Source: §3 · p. 5

3.1 Lamperti Transform for GBM

• The log Lamperti transform removes GBM’s state-dependent diffusion, enabling a reverse-time SDE with a state-independent diffusion coefficient.The score in log space is rewritten using the score change-of-variables relation.Source: §3.1 · pp. 5–6

3.2 Discretization of the Reverse-Time SDE

• Euler–Maruyama discretization yields two multiplicative samplers: the ambient-space sampler is sign-agnostic, while the log-space sampler preserves signs.The sign-preserving update uses a strictly positive exponential scaling factor and is named the Dale-Langevin sampler.Source: §3.2 · p. 6

• The multiplicative score term Xt ◦ ∇log pXt(Xt) motivates multiplicative explicit and denoising score-matching losses for training a score network.The marginal explicit loss is intractable because the marginal density is unavailable, whereas the conditional score from the forward process is tractable.Source: §4 · p. 7
• Under regularity and boundary assumptions, the multiplicative denoising and explicit score-matching losses are equivalent up to a parameter-independent constant.This permits optimization using the tractable conditional score rather than the unknown marginal score.Source: §4 · p. 7
• At t = 0, the multiplicative explicit loss recovers Hyvärinen’s score-matching loss for non-negative data.The paper frames this as a generalization of that static formulation to the multiplicative setting.Source: §4 · p. 7

5 Dale’s Law, Exponentiated Gradient Descent, and Mirrored Langevin Dynamics

• A shared convex generator unifies EGD and the Dale-Langevin sampler through their multiplicative exponential update structure.The figure presents this shared structure as a formal duality between biologically plausible learning and score-based sampling.Source: §5 · p. 7, Figure 2 · p. 8

5.1 Exponentiated Gradient Descent and Dale’s Law

• EGD respects Dale’s law and produces log-normal weights, whereas standard gradient descent permits synaptic flips and does not produce log-normal weights.EGD uses the negative-entropy mirror map; the paper also describes it as superior to gradient descent for synaptic pruning.Source: §5.1 · p. 8
• The multiplicative score term Xk ◦ ∇log pXt(Xk, kδ) plays the role of the gradient in EGD’s structurally equivalent update.The compared updates are sign-preserving, and the paper distinguishes Dale’s law on network weights from sampler behavior on the state vector.Source: §5.1 · pp. 8–9, §5 · p. 8

5.2 Connection to Mirrored Langevin Dynamics

• Using the same h that yields EGD, Mirrored Langevin Dynamics gives a sampling SDE on the positive orthant, recovered from GBM’s reverse-time SDE via the Lamperti transform.The paper reports convergence guarantees for the log-normal target in the appendix.Source: §5.2 · p. 9
• The Annealed-Dale-Langevin sampler generates samples shown for Fashion-MNIST, Kuzushiji-MNIST, and MNIST using forward-process class-average initialization.The figure refers to configurations listed in Table 3.Source: Figure 3 · p. 9

6 Experiments

• The evaluation covers MNIST, Fashion-MNIST, Kuzushiji-MNIST, and CIFAR-10, with positive pixel values and 1,000 forward discretization steps.The grayscale images are 28 × 28; CIFAR-10 images are 32 × 32 × 3.Source: §6 · p. 9

6.1 Score Network Architecture

• Grayscale datasets use an NCSN-Deeper encoder-decoder with conditional normalization, while CIFAR-10 uses DDPM++.The described grayscale network has five residual encoder stages and five conditional decoder refine blocks with skip connections.Source: §6.1 · p. 10

6.2 Noise Initialization Strategies

• The reverse process can start either from forward-diffused per-class mean images or from log-normal noise fitted to terminal forward samples.For CIFAR-10, the fitted log-space Gaussian parameters are estimated channel-wise; other datasets use a single-channel estimate.Source: §6.2 · p. 10

6.4 Evaluation Metrics

• Class-averaged initialization gives comparable or better FID and KID than log-normal initialization in general, supporting both multiplicative samplers as viable generative models.Table 1 reports scores for annealed Dale-Langevin and sign-agnostic samplers under both initialization strategies.Source: Table 1 · p. 11, §6.4 · p. 12

6.3 Sampling Algorithms

• Annealed Euler–Maruyama samplers geometrically decay noise by factor χ across levels (χ < 1), consistently improving quality; several terminal denoising steps improved CIFAR-10 samples, while grayscale datasets required none.Source: §6.3 · p. 11

7 Conclusions and Outlook

• The framework uses GBM, which preserves strict positivity and has log-normal marginals, to define multiplicative diffusion.The work derives sign-agnostic and sign-preserving samplers, the latter termed Dale-Langevin.Source: §7 · p. 12
• The Dale-Langevin sampler is equivalent via the Lamperti transform to additive Langevin dynamics on the log-transformed variable.The framework connects multiplicative-space diffusion with score-based diffusion on R^d.Source: §7 · p. 12
• The convex function driving exponential gradient descent dictates Dale-Langevin dynamics under the Mirrored Langevin Dynamics framework.This links the sampling dynamics to the optimization framework.Source: §7 · p. 12
• M-DSM is equivalent to M-ESM up to a constant, and non-negative score matching emerges as a special case at t = 0.The paper presents this as a generalization of score matching to the multiplicative setting.Source: §7 · p. 12
• The framework may be adapted for denoising and restoration when the forward noise is multiplicative rather than additive.The paper identifies exploring gamma noise, high-resolution synthesis, and non-image domains as future directions.Source: §7 · p. 12

Limitations

• Good performance requires substantial training data and computational resources, while noise schedule and learning-rate choices can affect performance.The authors note these constraints are common to deep generative models, not unique to their approach.Source: Limitations · p. 12

Broader Impact Statement

• The framework's potential applications include restoration with multiplicative noise and financial time series, while misuse could produce biased or misleading content.The authors also identify over-reliance on generated outputs in downstream decisions as a possible harm.Source: Broader Impact Statement · p. 12

A Log-normal Distribution

• A log-normal variable W has a Gaussian-distributed log, and the multivariate construction gives independent, identically distributed entries.The paper denotes the univariate distribution LN(µ, σ²) and the multivariate form LN(µ, σ²I).Source: A Log-normal Distribution · p. 17

B Equivalence Between Multiplicative Denoising Score-Matching and Multiplicative Explicit Score-Matching

• Under regularity and boundary conditions, M-DSM and M-ESM are equivalent up to a parameter-independent constant, enabling optimization with tractable conditional scores.The paper states that the true marginal scores are unavailable, whereas conditional scores can be computed from the forward SDE.Source: B Equivalence Between Multiplicative… · pp. 18–19

C Time-Varying Drift and Diffusion SDE

• The appendix generalizes the GBM framework to time-dependent drift and diffusion and derives corresponding forward and reverse-time SDEs.It describes Euler-Maruyama discretization of the forward process and a reverse-time SDE.Source: C Time-Varying Drift and Diffusion SDE · pp. 19–20

D Mirrored Langevin Dynamics for a Log-normal Target

• For a log-normal target, the limiting density obtained from GBM does not converge to the correct target density and is biased.The discussion relates this to discretization bias in Langevin dynamics.Source: D Mirrored Langevin Dynamics for a… · p. 20

E.1 Training Details

• CIFAR-10 training runs for 1.3 million iterations, compared with 300k for MNIST and 200k for Fashion-MNIST and Kuzushiji-MNIST.Training uses AdamW and is distributed across two NVIDIA RTX 4090 and two NVIDIA A6000 GPUs.Source: §E.1 · p. 21
• The optimal L, δ, and χ are selected by grid search to minimize FID and KID, with 1,024 samples drawn per configuration.Tables 3 and 4 report selected settings for log-normal-noise and class-averaged initialization configurations.Source: §E.1 · p. 21, Table 3 · p. 22, Table 4 · p. 22
• Table 2 reports estimated µ and σ parameters used for lognormal-noise initialization; for CIFAR-10, both estimates are given channel-wise.Source: Table 2 · p. 21

F Generated Samples

• Generated samples are diverse, resemble training data, and are noise-free, while Fashion-MNIST samples are less diverse and some classes are underrepresented.The authors say this is not evidence of mode collapse and suggest greater image complexity and variability as a possible explanation.Source: F Generated Samples · p. 22

G Evaluation Metrics for the Generated Images

• FID measures feature-space distribution distance, while KID uses a kernel and is described as less sensitive to outliers and more robust for small samples.Lower FID values indicate better quality.Source: G Evaluation Metrics for the Generated… · p. 22

G.1 Nearest neighbours

• Ten training-set neighbours are retrieved by Euclidean distance in pixel and InceptionV3-feature spaces; they are coherent but nonidentical, supporting the authors’ generalization check.The authors report that pixel-space neighbours were particularly interpretable for their chosen datasets.Source: §G.1 · p. 23
• Nearest-neighbour panels cover four datasets under lognormal-noise and class-average-forward initializations with the Dale-Langevin sampler.The captions identify CIFAR-10, Fashion-MNIST, Kuzushiji-MNIST, and MNIST panels.Source: Figure 4 · p. 23, Figure 5 · p. 24, Figure 6 · p. 24

G.2 Effects of annealing in generation of samples

• χ=1.0 produces diverse but noisy samples, χ=0.995 produces noiseless samples with mode collapse, and χ=0.9995 is a middle ground with less noise and less mode collapse.The study compares χ values across CIFAR-10, Fashion-MNIST, Kuzushiji-MNIST, and MNIST.Source: §G.2 · p. 31
• The ablation varies χ and sampler architecture while holding L and step-size δ constant across four datasets and two initialization strategies.Initializations are forward-noised class averages or lognormal noise whose parameters are estimated from forward-noised class averages.Source: §G.2 · p. 31
• Uncurated sample panels compare annealing factors across both sampler algorithms and both initialization strategies.The captions include class-average-forward and lognormal-noise initializations for Dale-Langevin and sign-agnostic sampling.Source: Figure 20 · p. 32, Figure 24 · p. 36, Figure 28 · p. 40