Subspace Levenberg Marquardt Algorithms in Training Neural Networks

1. Introduction

• Greater network capacity can help represent more complex regression mappings and classifiers, but increases the number of trainable parameters and the demands of optimization.The capacity needed for a prescribed approximation accuracy can depend on properties such as target-function dimensionality and smoothness.Source: §1 · p. 1
• Full LM’s memory and computational costs make it impractical as network size increases, despite its effectiveness on small- to medium-sized networks.SGD and Adam use less memory and cheaper iterations, but may depend strongly on hyperparameters and require many iterations, especially on poorly conditioned landscapes.Source: §1 · p. 1
• Subspace LM methods reduce the linear system’s size while retaining important curvature information, offering a potential compromise between first-order methods and full LM.KSLM constructs Krylov directions; HSLM combines multiple informative directions.Source: §1 · p. 2
• The study evaluates LM, KSLM, and HSLM against SGD and Adam on neural-network regression and classification, focusing on convergence, computational cost, and predictive performance.The stated aim is to assess whether subspace formulations extend LM’s practical advantages to larger optimization problems.Source: §1 · p. 2

2.1. Levenberg–Marquardt Algorithm

• LM computes a damped Gauss–Newton update, with the damping parameter adjusted each iteration to shift behavior between Gauss–Newton and a scaled gradient-descent direction.The damping improves robustness when the Jacobian is ill-conditioned or the current iterate is far from a solution.Source: §2.1 · p. 2

2.2. Krylov-Subspace Levenberg–Marquardt Algorithm

• KSLM restricts the step to a Krylov subspace and solves a smaller projected system instead of the full n-dimensional LM system.For fixed Jacobian and gradient, its basis can be reused while adjusting the damping parameter, avoiding recomputation.Source: §2.2 · p. 3

2.3. Hybrid Subspace Levenberg-Marquardt Algorithm

• HSLM builds its low-dimensional basis from gradient, recent-step, Krylov/Lanczos, and randomized Gauss–Newton probe information.An adequacy test checks captured gradient energy and adds candidate directions when the measure falls below a prescribed threshold.Source: §2.3 · pp. 3–4
• HSLM replaces isotropic damping with a diagonal curvature-dependent matrix and uses Armijo backtracking for step acceptance.The method adjusts damping according to whether the step is accepted or rejected.Source: §2.3 · p. 4

3.1. Damped Oscillation Regression Problem

• All five methods approximated the damped-oscillation function, but differed in convergence speed and computational cost.SGD and Adam averaged 155.0 ± 55.1 and 102.1 ± 44.3 epochs before termination, respectively.Source: §3.1 · pp. 4–5, Figure 1 · p. 5
• For damped-oscillation regression, HSLM averaged 19.3 ± 5.2 iterations and 25.9 ± 7.4 s, versus 22.2 ± 6.8 iterations for LM and KSLM, which took 83.0 ± 26.2 s and 132.8 ± 40.5 s, respectively.All three LM-based methods reached the prescribed noise-variance threshold in 100% of trials, versus 41% for SGD and 0% for Adam.Source: Table 1 · p. 5, §3.1 · p. 5

3.2. 13-Parity Classification Problem

• On 13-bit parity, LM-based methods reduced training and validation MSE faster than SGD and Adam, while all five methods achieved high training and validation accuracy.HSLM and Adam converged in every trial; Adam had the highest validation accuracy, while HSLM was comparable with far fewer iterations.Source: §3.2 · pp. 5–6, Figure 2 · p. 6
• On 13-bit parity, HSLM averaged 37.0 ± 27.2 outer iterations and reached 99.6 ± 0.3% validation accuracy, compared with Adam’s 99.8 ± 0.1%.HSLM and Adam each had a 100% convergence rate; LM, KSLM, and SGD had rates of 92%, 89%, and 92%, respectively.Source: §3.2 · pp. 5–6, Table 2 · p. 6

4. Conclusion and Future Work

• HSLM achieves a favorable balance of convergence speed, computational efficiency, and solution accuracy on the evaluated regression and classification problems.The conclusion suggests reduced-subspace approaches can preserve fast LM-type convergence while reducing computational cost for neural-network training.Source: §4 · p. 6
• Larger datasets may increase the cost of forming and solving subproblems, limiting scalability.Source: §4 · p. 6
• Future work may explore mini-batch and matrix-free implementations, broader subspace methods, and how basis choices affect efficiency and convergence.Source: §4 · p. 6

Appendix A. Experimental Setup

• Five methods—SGD, Adam, classical LM, KSLM, and HSLM—were tested on nonlinear regression and 13-bit parity classification.Source: Appendix A · p. 9
• Each experiment used 100 independent trials, with parameters initialized from θ0 ∼U[−1, 1] and the same initial vector for all methods within each trial.Source: Appendix A · p. 9
• Experiments used Python and NumPy in Jupyter Notebook on a Dell 14 Plus laptop with an Intel Core Ultra 9 288V processor and 32 GB RAM.Source: Appendix A · p. 9

A.1. Regression problem

• The regression task approximates a target function using 40,000 inputs with additive Gaussian noise and a 1−70−40−1 MLP.The noise standard deviation is 5% of the target standard deviation; the generated data and noise realization are fixed across optimizers and trials.Source: §A.1 · p. 9

A.2. 13-bit parity problem

• The classification task uses all 8,192 patterns of 13-bit parity and a 13−25−10−1 MLP.Inputs and labels use {−1, 1}; the fixed stratified split assigns 90% of the data to training and 10% to validation.Source: §A.2 · p. 9

A.3. First-order methods

• SGD uses a 0.01 learning rate and 0.9 momentum, while Adam uses a 0.003 learning rate and β1 = 0.9, β2 = 0.999.Both use mini-batches of 64 and are allowed up to 1,500 epochs.Source: §A.3 · p. 9

A.4. LM-based methods

• LM, KSLM, and HSLM use up to 150 outer iterations and begin with damping parameter µ0 = 10.The delayed-gratification strategy halves damping after a successful step and multiplies it by five after an unsuccessful step.Source: §A.4 · pp. 9–10
• KSLM limits its Krylov subspace to 5% of parameters, while HSLM adapts its subspace up to 10% of the parameter dimension.HSLM starts with 1% random probes, uses an adequacy threshold of 0.99, and can add Lanczos and random Hessian-probe directions.Source: §A.4 · p. 10

A.5. Stopping criteria

• Regression stopping uses a noise-variance training-error target, while parity requires MSE ≤ 10^-2 and training accuracy above 99%.SGD and Adam can also stop after 50 epochs with relative MSE improvement below 10^-5; LM methods have gradient- and step-size stopping conditions.Source: §A.5 · p. 10
• For each optimizer, the study records training and validation MSE, classification accuracy, iterations or epochs, and execution time.Source: §A.5 · p. 10

B.1. Damped Oscillation Regression Problem

• Figure 3 compares function approximations and plots training MSE against epochs or iterations for five optimization methods.Source: Figure 3 · p. 10, §B.1 · p. 10

B.2. 13-Parity Classification Problem

• Figure 4 separates 13-parity training and validation results into MSE and accuracy panels.Source: Figure 4 · p. 11

• 30/70 training/validation split was used to evaluate optimizer performance under limited-data conditions on the 13-bit parity problem.Source: Appendix C · p. 11
• HSLM maintained a stronger balance of convergence speed, computational efficiency, and predictive accuracy than other methods in the limited-data parity experiment.Source: Appendix C · p. 11, Figure 5 · p. 12
• Figure 5 separates training and validation MSE and accuracy into four panels for the 13-parity classification problem.Source: Figure 5 · p. 12