Connecting Score Matching, Maximum Likelihood, and Expectation-Maximization in Mixed Linear Regression

1 Introduction

• The study focuses on MLR because a multimodal response mean can lie in a low-probability region, while conditional diffusion models learn scores of noise-smoothed responses.The model has covariate-dependent component centers and unknown mixing weights.Source: §1 · pp. 1–2

1.1 Technical Overview and Contributions

• Controlling the terminal discrepancy transfers consistency and inverse-Fisher asymptotic normality from maximum likelihood to the path-integrated score-matching estimator.The terminal population KL divergence preserves identifiability.Source: §1.1 · p. 2
• Fixed-scale score-matching gradients involve EM operators, operator derivatives, and latent-variance terms, with distinct low- and high-noise endpoint behavior.The loss has equivalent cross-entropy and EM-operator decompositions.Source: §1.1 · pp. 2–3
• O(K^-2) loss bounds hold when the effective target is zero; generic initializations with a nonzero target have O(log(1/ϵ)) bounds, except for a measure-zero saddle-attracted set.Along fixed rays of diverging ground-truth signal, the imbalance-parameter gradient vanishes at fixed diffusion noise.Source: §1.1 · p. 3

2 Problem Setup

• The setup specifies the MLR model, variance-preserving response diffusion, population and empirical score-matching objectives, and associated likelihood and EM operators.It also introduces notation for the estimation and asymptotic analysis.Source: §2 · p. 3

2.1 Mixed Linear Regression and Variance-Preserving Diffusion

• The two-component model uses positive mixing weights summing to one, independent covariates and latent assignments, unit-variance Gaussian noise, and positive-definite covariate covariance.The half log-odds parameter ν represents the mixing imbalance.Source: §2.1 · p. 3

2.2 Score Function and Score Matching Loss

• The score-matching loss compares fitted and ground-truth conditional scores through relative Fisher information, while the Stein score is given explicitly for the diffused response.The score formula is s_{θ_t,ν}(y_t,x) = -y_t + tanh(μ_t y_t + ν)μ_t.Source: §2.2 · p. 4

2.3 Negative Log-Likelihood and Expectation-Maximization Operators

• The cross-entropy is the expected negative log-likelihood under the ground-truth law, and the section defines population EM operators for regression and mixing imbalance.Source: §2.3 · p. 4

2.4 Notations on Metrics, Asymptotics, and Estimation Objectives

• The notation distinguishes the empirical negative log-likelihood, path-integrated score-matching objective, terminal discrepancy, and terminal-horizon constant.The terminal discrepancy is an empirical conditional KL divergence and is nonnegative.Source: §2.4 · p. 5

3 Path-Integrated Score Matching: Maximum-Likelihood Asymptotics

• The path-integrated score-matching estimator inherits consistency and asymptotic normality with inverse-Fisher covariance when the terminal discrepancy is controlled.The terminal population KL divergence preserves identifiability.Source: §3 · p. 5
• The integrated score-matching loss equals an initial joint KL divergence minus a terminal joint KL divergence.Source: §3 · p. 5
• The terminal population KL is bounded by a factor proportional to ¯α_T^2/(1−¯α_T) times the expected squared Wasserstein distance between the two-point mixing measures.The empirical terminal discrepancy decays at the same stated rate in ¯α_Tn as sample size grows.Source: §3 · p. 5

• Under the terminal schedule condition, the terminal discrepancy is oP(n^-1) for bounded parameters, enabling transfer of maximum-likelihood guarantees.Theorem 3.2 assumes θ* ≠ 0 and a fourth-moment condition for asymptotic normality.Source: Appendix A · pp. 5–6
• The terminal population KL divergence has exactly two global minimizers, (θ*, ν*) and (−θ*, −ν*).The stated scope assumes retained signal, θ* ≠ 0, and covariates with a density positive almost everywhere and finite second moments.Source: Appendix A · p. 5
• The likelihood’s invariance under (θ, ν) ↦ (−θ, −ν) makes joint sign alignment necessary when expressing parameter consistency.Source: Appendix A · p. 6
• After joint sign alignment, the path-integrated estimator is consistent and has the MLE’s n^-1/2 Gaussian limit with covariance I(θ*, ν*)^-1.The limit follows from the local likelihood expansion under the stated fourth-moment condition.Source: Appendix A · p. 6

4 Fixed-Scale Score Matching: Cross-Entropy and EM Decompositions

• The fixed-scale score-matching loss has equivalent decompositions using cross-entropy and EM operators, yielding exact gradient identities involving operator derivatives and latent variance.The section also relates the diffused MLR operators to cross-entropy gradients.Source: §4 · pp. 7–8
• At low noise, the gradient expansion retains EM-related terms, operator-Jacobian corrections, and latent-variance corrections, with an O(1 − alpha_bar_t) remainder.The expansion assumes E[||x||^4] < ∞; the objective limit itself requires only a second moment.Source: §4 · pp. 8–9
• At high noise, the limiting objective depends on the candidate and ground-truth parameters through the effective coefficient c* = θ* tanh ν*, with an O(alpha_bar_t) gradient remainder.The limit assumes a finite second moment, and the stated gradient remainder is supported under a fourth-moment condition.Source: §4 · p. 9
• The leading high-noise objective is flat in θ when ν* = ν = 0, so it supplies no preferred θ direction on that balanced slice.The paper treats a separate next-order directional diagnostic at small positive noise.Source: §4 · p. 9
• The endpoint gradient remainders are illustrated componentwise at one fixed model pair, with Figure 2 comparing low- and high-noise expressions.Source: Figure 2 · p. 9

5.1 High-Noise Gradient Dynamics

• For isotropic covariance, admissible fixed-step descent has O(K^-2) loss when c* = 0 and reaches any ε > 0 in O(log(1/ε)) iterations for generic nonzero-target initializations.The exceptional saddle basin has measure zero; its trajectories converge to the saddle manifold and retain limiting loss ||c*||^2/2.Source: §5.1 · p. 10
• The high-noise objective identifies ground truth only through c* = θ* tanh ν*, and zero-target trajectories may decay faster than the uniform K^-2 upper bound.For nonzero c*, the transient and contraction factor depend on initialization.Source: §5.1 · pp. 10–11

5.2 Score Matching Blindness at High SNR

• At fixed model pair and noise scale, as ||θ*|| diverges along a fixed ray, both |∇νL_t| and V_t tend to zero pointwise.The proposition states a finite-second-moment condition on covariates and, for nonzero θ, an additional projection condition.Source: §5.2 · p. 11
• The vanishing imbalance gradient removes the local first-order score-matching signal for changing candidate mixing weights, while the EM update can retain true-imbalance information through regression alignment.The accompanying latent-variance limit records disappearance of uncertainty about component assignment.Source: §5.2 · p. 11

6 Numerical Experiments

• Under the theorem’s terminal schedule and joint sign alignment, Figure 1 illustrates the integrated estimator’s predicted parametric scaling and joint calibration.Source: §6 · p. 11
• At a fixed model pair, Figure 2 checks componentwise low- and high-noise gradient-remainder orders using their respective endpoint expressions.The low-noise expression includes the EM operator and latent variance; the high-noise expression uses the effective coefficient.Source: §6 · p. 11
• For displayed admissible initializations, Figure 3 illustrates the zero-target polynomial upper bound, generic geometric decay, and exceptional saddle described by Theorem 5.1.Source: §6 · p. 11
• In a one-dimensional specialization, Figure 4 shows the imbalance gradient and posterior-variance term decreasing along a fixed-candidate, fixed-scale ray.This illustrates Proposition 5.2’s pointwise conclusion at one candidate and scale.Source: §6 · p. 11

7 Conclusions

• The path-integrated estimator recovers regression parameters and mixing weights up to joint-sign symmetry, with inverse-Fisher asymptotic normality matching maximum likelihood.Under the stated identifiability conditions, the terminal population KL divergence has the ground truth as its global minimizer at every positive retained-signal level.Source: §7 · p. 12
• The fixed-scale score-matching loss admits cross-entropy and EM-related decompositions, and its gradient includes operator-Jacobian and latent-variance terms.At low noise, the limiting gradient retains the EM-related correction structure.Source: §7 · p. 12
• Under isotropic covariance, the high-noise limiting loss has an O(K^-2) loss bound for a zero effective target and eventual geometric decay for generic initializations otherwise.Only a measure-zero exceptional basin is attracted to the saddle manifold.Source: §7 · p. 12
• Along fixed high-SNR ground-truth rays, the imbalance gradient and latent-variance term converge pointwise to zero at covered fixed candidates and diffusion scales.This removes the local first-order signal for changing candidate mixing weights.Source: §7 · p. 12

Appendix B

• Appendix B proves consistency up to joint sign, the aligned Gaussian limit with inverse-Fisher covariance, and high-probability attainment of the empirical score-matching minimum.Source: E Experimental Details and Additional… · p. 18

Appendix C

• Appendix C proves the cross-entropy–activation and EM–latent-variance decompositions, their gradient identities, and the low- and high-noise endpoint results.Source: E Experimental Details and Additional… · p. 18

Appendix D

• Appendix D establishes high-noise gradient convergence rates, pointwise score-matching blindness along fixed high-SNR rays, and a hard-assignment EM comparison.Source: E Experimental Details and Additional… · p. 18

Lemmas A.1 and A.3

• The diffusion preserves standard-Gaussian noise, and the conditional Stein score is −y_t + μ_t tanh(μ_t y_t + ν).The score uses the scaled mean μ_t and the mixing-weight imbalance parameter ν.Source: A Auxiliary Identities and Bounds for… · pp. 19–20

Lemmas A.4 and A.5

• Lemmas A.4–A.5 derive conditional KL dissipation and relate the log-likelihood derivative to the empirical score-matching loss.Source: A Auxiliary Identities and Bounds for… · p. 20

Lemmas A.6 and A.7

• For 0 < ᾱ_t < 1, the diffused conditional KL is bounded by ᾱ_t/[2(1−ᾱ_t)] times the squared initial-law Wasserstein-2 distance, which is itself bounded by the distance between latent discrete laws.This combines the decay bound with the Wasserstein contraction under Gaussian smoothing.Source: A Auxiliary Identities and Bounds for… · p. 22

Lemma A.8

• For a fixed initial value y_0, the transition-kernel KL is bounded by ᾱ_t/[2(1−ᾱ_t)] times y_0^2 + μ_0^2 − 2y_0μ_0 tanh(ν) + 1.Source: A Auxiliary Identities and Bounds for… · p. 22

Lemma A.10

• As ᾱ_t tends to zero, the scaled score-matching losses converge to squared differences of imbalance-weighted means or to an expected squared difference of the corresponding tanh score terms.The lemma assumes data generated under the ground-truth parameters; the limits are established using dominated convergence.Source: A Auxiliary Identities and Bounds for… · p. 24

Wasserstein identities

• The appendix gives an explicit squared Wasserstein-2 identity for diffused MLR distributions and a Gaussian specialization based on correlated Gaussian projection moments.The general identity assumes isotropic second moments for x; the Gaussian specialization treats nonzero parameter vectors.Source: A Auxiliary Identities and Bounds for… · p. 25

Terminal KL characterization

• Under the stated positive-density and nonzero-truth conditions, terminal KL divergence is zero exactly at the ground-truth parameters and their joint sign-flipped equivalent.The joint sign change preserves the terminal conditional distribution by exchanging the mixture components.Source: A Auxiliary Identities and Bounds for… · p. 26

Integrated score-matching identity

• The integrated relative de Bruijn identity relates initial conditional KL to terminal KL plus score matching loss along the diffusion path.It applies when x is independent of the latent variables and the diffusion schedule has positive signal level.Source: A Auxiliary Identities and Bounds for… · p. 26

Finite-sample likelihood identity

• For finite samples and an interior terminal noise level, negative log-likelihood decomposes into a terminal denoising KL term and the empirical auxiliary score-matching loss.The stated condition is T > 0 and 0 < ᾱ_T < 1.Source: A Auxiliary Identities and Bounds for… · p. 26

Likelihood derivatives

• Closed-form derivatives of the conditional negative log-likelihood are provided with respect to the response, projected mean, and mixture parameter.The response and projected-mean derivatives involve tanh(μ_t y_t + ν).Source: A Auxiliary Identities and Bounds for… · p. 27

Spiked covariance: Gaussian

• For Gaussian covariates, the whitened spiked covariance has eigenvalue 3c along the transformed truth direction and c on its orthogonal complement, yielding generalized spectral gap 2c.Here c = (θ*)^TΣθ*; the leading generalized eigenspace is span{θ*}.Source: A Auxiliary Identities and Bounds for… · p. 28

Spiked covariance: elliptical

• For zero-mean elliptical covariates with finite fourth moments, excess kurtosis scales the corresponding eigenvalues to 3(1 + κ)c and (1 + κ)c, with gap 2(1 + κ)c.The leading generalized eigenspace remains span{θ*}.Source: A Auxiliary Identities and Bounds for… · p. 28

Proof of Main Theorem 3.2

• Under the stated horizon and covariate conditions, the estimator is consistent up to joint sign; with finite fourth covariate moment, its aligned error has a Gaussian limit with inverse-Fisher covariance.The theorem’s schedule condition is ln(1/(1−ᾱ_Tn))−ln n→∞.Source: B Proofs of Maximum-Likelihood… · p. 29
• The population conditional likelihood identifies the ground truth only up to joint sign: its global maximizers are (θ*, ν*) and −(θ*, ν*).Positive-almost-everywhere covariate density and θ* ≠ 0 yield this identifiability under the joint sign symmetry.Source: B Proofs of Maximum-Likelihood… · pp. 31–32
• The empirical bridge and score-matching optimality give likelihood suboptimality oP(n^-1), supplying the near-optimality condition used for consistency and the aligned Gaussian limit.The proof applies Wald’s consistency theorem and then the local M-estimator asymptotic-normality theorem.Source: B Proofs of Maximum-Likelihood… · pp. 32, 34

Lemma B.2: High-probability attainment of the empirical minimum

• When d is fixed and the stated model, covariate, and terminal-horizon conditions hold, the empirical score-matching objective attains its minimum with probability tending to one, with a sample-measurable exact minimizer selectable.The horizon condition is ᾱ_Tn/(1−ᾱ_Tn)=o(n^-1).Source: B Proofs of Maximum-Likelihood… · p. 34

Boundary gap and finite minimizers

• The boundary gap equals an expected conditional KL divergence and is strictly positive because the true conditional variance exceeds the boundary Gaussian’s unit variance almost everywhere.The true variance is 1 + sech^2(ν*) (u*)^2, while the boundary Gaussian has variance one.Source: B Proofs of Maximum-Likelihood… · p. 35
• On the stated event, every compactified minimizer has |ρ| < 1 and maps through ν = arctanh ρ to an exact finite minimizer.The event is measurable, and a measurable selector can be constructed by minimizing |ρ| over the compact argmin correspondence and lifting fitted values through X+.Source: B Proofs of Maximum-Likelihood… · p. 35

Fixed-Sample Nonattainment

• For a fixed sample, finite-parameter attainment can fail: the scalar example has an empty finite-parameter argmin even though boundary choices attain the negative lower bound.The obstruction has vanishing probability in the stated asymptotic regime.Source: B Proofs of Maximum-Likelihood… · p. 35

C Proofs of Cross-Entropy and EM Decompositions for Fixed-Scale Score Matching

• The appendix restates and proves the fixed-scale cross-entropy and EM decompositions from Section 4.Source: C Proofs of Cross-Entropy and EM… · p. 36
• The regression-parameter EM update is M(θt, ν) = Σ−1E∗[xyt tanh(µtyt + ν)].Source: C Proofs of Cross-Entropy and EM… · p. 36
• The imbalance-parameter EM update is N(θt, ν) = E∗[tanh(µtyt + ν)].Source: C Proofs of Cross-Entropy and EM… · p. 36

Their derivatives satisfy

• The cross-entropy gradients equal the EM residuals Σ(θt − M(θt, ν)) and tanh ν − N(θt, ν).Source: Their derivatives satisfy · p. 36
• The fixed-scale score-matching loss has a latent-variance representation and an equivalent activation form.Source: Their derivatives satisfy · p. 37
• Normalized EM operators rescale M and N at diffusion time t, with Vt defined as V(θt, ν)/¯αt.Source: Their derivatives satisfy · p. 38
• As ¯αt → 1, ∇θLt approaches Σ(θ − M) − (∇θM)⊤Σθ + ∇θV, with an O(1 − ¯αt) remainder when E[∥x∥4] < ∞.Source: Their derivatives satisfy · p. 38
• As ¯αt → 0, Lt converges to 1/2∥θ tanh ν − θ∗tanh ν∗∥²Σ when E[∥x∥2] < ∞.Source: Their derivatives satisfy · p. 39

Appendix scope

• The appendix also proves the complementary high-SNR blindness result and a companion hard-assignment limit supporting comparison with population EM.The supplied passages identify these proof contributions without specifying their detailed statements.Source: D Proofs for High-Noise Gradient… · p. 40

High-noise gradient dynamics

• The admissible set Dη is forward invariant, and the loss is nonincreasing along every gradient-descent trajectory initialized there.The proof also establishes preservation of admissibility at each iterate.Source: D Proofs for High-Noise Gradient… · pp. 40, 42
• The exceptional initializations converging to S form a measure-zero set and have limiting loss ∥c*∥²/2.For c* = 0, a separate sublinear loss bound holds.Source: D Proofs for High-Noise Gradient… · pp. 40, 43
• For c* ≠ 0, generic admissible initializations reach loss ε in O_{ϑ0,η,c*}(log(1/ε)) iterations; the bound is pointwise, not uniform in the entry time.Generic initializations exclude the stable set of the saddle set S.Source: D Proofs for High-Noise Gradient… · pp. 40, 45
• Every point on S is fixed by the update, while a boundary initialization outside S enters the matching-sign region after one update.These results distinguish saddle-set initializations from trajectories that proceed toward generic convergence.Source: D Proofs for High-Noise Gradient… · p. 43

D Proofs for High-Noise Gradient Dynamics and Score Matching Blindness

• At a fixed model pair and diffusion-noise scale, |∇νLt| and Vt both tend to zero as the ground-truth signal norm grows along a fixed ray.The result assumes E[∥x∥²] < ∞, with the ground-truth mixing weights and other model and data-generating quantities fixed.Source: D Proofs for High-Noise Gradient… · p. 46
• For isotropic Gaussian covariates, q(θ, θ*) = (2/π) arcsin ρ, where ρ is the cosine similarity of θ and θ*.The result follows from q(θ, θ*) = E_x[sgn(⟨θ, x⟩⟨θ*, x⟩)].Source: D Proofs for High-Noise Gradient… · p. 47
• Along the same fixed-ray limit, the EM imbalance operator converges to q(θ, θ*) tanh ν*, while its derivative with respect to ν tends to zero.Here q(θ, θ*) = E_x[sgn(⟨θ, x⟩⟨θ*, x⟩)] is constant along the ray.Source: D Proofs for High-Noise Gradient… · p. 47

Finite-sample estimation and calibration

• Table 4 reports joint 95% Fisher-ellipse coverage and relative covariance discrepancy, and all 1,000 optimization runs terminated without solver failure.The covariance discrepancy compares the empirical covariance of scaled aligned errors with inverse Fisher information.Source: Table 4 · p. 49, E Experimental Details and Additional… · p. 49

Endpoint-gradient checks

• The low-noise endpoint reference differed from the complete operator expression by 2.24964 × 10−9 in Euclidean norm, confirming inclusion of correction terms.The high-noise reference is the gradient of the effective-product limiting loss.Source: E Experimental Details and Additional… · p. 50

Spiked fourth-order covariance and local spectral diagnostic

• Whitening identifies span{θ∗} as the leading generalized eigenspace, with generalized spectral gap 2σ2µ in the elliptical case.For isotropic covariance, the elliptical fourth-order covariance is (1+κ) times its Gaussian counterpart.Source: E Experimental Details and Additional… · p. 50
• On the balanced slice, the leading high-noise objective is flat in θ; Figure 5 instead compares finite-noise update directions with normalized power steps.The directional comparison does not assert equality of unnormalized updates or follow from the proposition’s O(ᾱt) bound.Source: E Experimental Details and Additional… · pp. 50–51, Figure 5 · p. 51

Limiting-dynamics diagnostics

• The degenerate-path fit had slope −2 −1.5035 × 10−2, while displayed limiting-dynamics paths showed no one-step loss increase.The slope fit is a finite-window diagnostic; the paths were consistent with the theorem’s O(K−2) bound and monotonicity conclusions.Source: E Experimental Details and Additional… · p. 51

Pointwise high-SNR blindness

• The high-SNR computation sums over latent classes, checks response-noise quadrature with paired 80- and 160-node rules, and adaptively integrates in the boundary-layer coordinate.Reported scaled components are rejected if the paired-order difference exceeds 10−8 + 10−6 times the larger absolute result.Source: E Experimental Details and Additional… · p. 52
• 1.6688 × 10−3 and 2.9399 × 10−9 were the computed absolute imbalance-gradient and latent-variance values at the largest displayed signal norm.The high-SNR experiment uses a scalar Gaussian covariate and fixed positive ground-truth direction.Source: E Experimental Details and Additional… · p. 52
• 0.3023024610 was the moving-candidate gradient limit, while the fixed-candidate cross-entropy gradient approached 0.836896 in magnitude.The moving-candidate curve illustrates pointwise-in-candidate scope; the cross-entropy control retains imbalance signal.Source: E Experimental Details and Additional… · pp. 52–53