Blockwise Stabilized Adaptive Cubic Regularization with Subsolvers via Recurrence

1 Introduction

• First-order methods exhibit spectral bias because gradient updates fit low frequencies before high frequencies, motivating second-order optimization for INRs.The experiments focus on INRs, although the introduced optimizer variants have broader application scope.Source: §1 · p. 1
• Existing full-Hessian cubic methods become impractical beyond roughly 15k–20k parameters, while original ARC stalls at 4.0 dB on the 91.4M-parameter ViSIR benchmark despite matrix-free Lanczos.The introduction attributes improved scalability to blockwise organization rather than Krylov solving alone.Source: §1 · pp. 2–3

1.1 Contributions

• The optimizer partitions parameters by tensor, assigning each block its own adaptive regularization constant, ratio test, and monotone full-loss acceptance decision.This blockwise regularization differs from the ARC-based families reviewed in Section 7.Source: §1 · p. 2, §1.1 · p. 3
• Four interchangeable subsolvers provide exact cubic Krylov steps, φ1 trust-controlled exponential relaxation, or Chebyshev recurrence updates with solver-specific termination criteria.CubicKrylov and ARC-Block share the exact cubic step; ARC-φ1 includes clamped saddle escape along negative modes.Source: §1.1 · p. 3, Table 1 · p. 4
• The theory proves exact cubic-model minimization within the Krylov subspace, O(ε^-3/2) iteration complexity, approximate second-order convergence, and monotone per-block descent.The exact subspace reduction is established in Lemma 3.5, with decrease and gradient bounds supporting the complexity result.Source: §1.1 · p. 3, §1 · p. 3
• At 91.4M parameters, ARC-Block and the per-block ARC control are the only evaluated cubic Newton methods whose steps remain exact on every block.The full true block Hessian is accessed through HVPs, including an 88.5M-parameter decoder tensor containing 97% of the model.Source: §1.1 · p. 3, §1 · p. 3
• 133.5 dB: ARC-φ1 reaches this FINER full-convergence PSNR versus 78.2 dB for tuned Adam’s extended-budget ceiling.CubicKrylov also reaches 124.7 dB versus 65.2 dB for converged tuned Adam.Source: §1.1 · p. 3

2.1 Problem setting

• The analysis assumes a twice-differentiable, generally nonconvex neural-network loss with Lipschitz Hessian and uses graded nonconvexity to control negative curvature.These assumptions support the standard cubic upper bound and bound the cubic shift required for positive definiteness.Source: §2.1 · p. 4

2.2 Chebyshev second kind preconditioning

• Chebyshev second-kind recurrence uses Hessian-vector products to construct a polynomial preconditioner whose worst-case residual contracts to about 0.22 per sweep.The recurrence requires one Hessian-vector product per degree and O(1) auxiliary vectors, while equalizing contraction across frequencies.Source: §2.2 · pp. 4–5
• The Chebyshev construction incurs degree L ≈ 1.28√κ and can grow hyperbolically for negative eigenvalues, requiring clamping safeguards.The degree bound depends on the Hessian condition number, while indefiniteness provides saddle escape but introduces failure modes.Source: §2.2 · p. 5

2.3 Cubic regularization as a shifted linear solve

• Cubic regularization is a shifted Newton solve; this work replaces Hessian diagonalization on large blocks with an adaptive Krylov polynomial built and regularized independently per parameter tensor.Unlike prior whole-space Krylov cubic regularization, the method targets nonconvex second-order stationarity, adapts each block’s cubic constant, and ties polynomial degree to the cubic shift.Source: §2.3 · pp. 5–6

3.1 The secular equation and the spectral shift

• The cubic model has a unique admissible secular solution whose shifted Hessian is positive definite, making the resulting step its global minimizer.The construction removes the need to clamp an indefinite spectral interval before applying the polynomial solver.Source: §3.1 · pp. 6–7

3.2 The shift bounds the Chebyshev degree that sets the Krylov subspace dimension

• The cubic shift controls the effective condition number and Chebyshev degree, while near a strict second-order point it vanishes and recovers undamped Chebyshev–Newton behavior.The local polynomial residual contracts by at most 0.22 per degree-L application, whereas far from stationarity the required degree can be constant.Source: §3.2 · pp. 7–8

3.3 Krylov subspace solver and its optimality

• Lanczos followed by a reduced secular solve returns the exact minimizer of the cubic model over the generated Krylov subspace.This stored-basis step uses O(n_bL) memory, unlike the fixed-coefficient recurrence’s O(1) vector storage, and an optional eigenvector augmentation handles the hard case.Source: §3.3 · p. 9

3.4 One-step progress and global complexity

• Exact cubic steps yield monotone descent, O(ε^-3/2) global complexity, and an iterate satisfying approximate first- and second-order stationarity.The curvature threshold is proportional to √(L_H ε), so strict saddles are excluded as stopping points.Source: §3.4 · pp. 10–11
• Krylov-inexact steps retain the global and second-order complexity guarantees under ARC termination criteria, while multiplicative per-block adaptation stabilizes unknown cubic constants.The adaptation doubles or quadruples constants after rejected steps and halves them after accepted steps, with rejected steps costing at most one extra function evaluation each.Source: §3.4 · p. 11
• Approximate second-order stationarity is the strongest guarantee on the studied nonconvex class: O(ε^-3/2) oracle complexity is optimal, while global guarantees require additional gradient-dominance structure.Sinusoidal INR landscapes are not gradient dominated because their saddle structure violates the assumptions that make every stationary point globally minimizing.Source: §3.4 · p. 11

3.5 Block-diagonal decomposition at scale

• Sequentially minimizing each tensor block’s cubic model with fresh gradients and accepting only non-increasing full-loss updates yields monotone iterations and per-block descent when M_b ≥ L_H,b.The block model uses the true diagonal Hessian block H_b and a block-specific cubic constant M_b.Source: §3.5 · p. 12
• With B ≈40 large tensors, joint acceptance rejects nearly every step and drives shared M to M_max, whereas per-block acceptance avoids this stall at one extra forward pass per block.Joint tests omit cross-block Hessian couplings and require inflation of each M_b by Θ(B); per-block acceptance uses separate constants and tests.Source: §3.5 · p. 12

4.1 The CubicKrylov step (single block)

• The CubicKrylov step builds a Lanczos basis with L Hessian-vector products, then solves the cubic model on the resulting (L+1) × (L+1) reduced matrix.All secular-search trial shifts reuse the cached basis, avoiding block-Hessian eigendecomposition and keeping the root-finding loop at L HVPs total.Source: §4.1 · pp. 12–13

4 ARC-Block Algorithm

• ARC-Block applies unconditional lazy exact cubic steps to small blocks and per-block accepted CubicKrylov steps to large blocks, with adaptive per-block regularization.Small blocks are evaluated against the post-phase-A loss; large-block steps are rejected on loss increases or nonfinite values, while accepted and rejected steps scale M_b by 1/2 and 4, respectively.Source: §4.1 · p. 13, §4.2 · p. 13

4.2 Cubic Newton in Krylov subspace on large blocks

• The large-block design uses unconditional small-block updates, per-block acceptance tests, fresh phase-B gradients, and a near-zero M_b floor to prevent vetoes and silent stalls.On nano-ViSIR, reusing a stale step-start gradient reduced final PSNR by approximately 8.5 dB, from 31.7 to 23.1 dB.Source: §4.2 · pp. 13–14

4.3 Cost accounting

• For L approximately 10, Krylov oracle cost is independent of block size n_b, while wall-clock time and basis memory scale linearly with n_b through HVPs and stored vectors.This enables cubic regularized second-order steps on tensors where only HVPs are affordable, including the 91.4M-parameter ViSIR SIREN decoder.Source: §4.3 · p. 14, Table 2 · p. 14

5 The Exponential-Relaxation (φ1) Step Subsolver

• ARC-φ1 uses the exact φ1 solution of the local quadratic gradient flow, integrated over horizon h, as an interchangeable trial-step rule.The step is the continuous-time limit of gradient descent on the frozen quadratic, integrated in closed form.Source: §5 · pp. 14–15

5.1 The φ1 step rule

• The φ1 multiplier approaches Newton steps on stiff positive modes, gradient steps near zero curvature, and exponentially amplifies negative modes, requiring an explicit clamp.Unlike the cubic multiplier, it separates flat, stiff, and negative modes without shift estimation or a secular solve, but lacks cubic-model step optimality.Source: §5.1 · pp. 15–16, §5.2.2 · p. 17

5.2.1 Krylov evaluation

• Krylov evaluation applies the φ1 matrix function in a Lanczos subspace with the same Hessian–vector-product budget as CubicKrylov at equal depth.Trust tightening reduces h_b∥H_b∥, making fixed-depth approximation easier, while outer acceptance absorbs subspace truncation.Source: §5.2.1 · p. 16
• The φ1 step guarantees positive quadratic decrease and gradient contraction for any spectrum, with positive-definite blocks converging toward Newton at an exponential rate.Its decrease is bounded below by 1/4 ∥g∥^2 min(h, 1/∥G∥), while positive-definite modes contract by e^-hλ_i.Source: §5.2.1 · p. 16

5.2.3 Trust-controlled horizons and inherited guarantees

• ARC-φ1 retains monotone guarded acceptance and trust adaptation, but has only O(ε^-2) accepted-step first-order complexity rather than cubic regularization’s O(ε^-3/2) rate.Second-order escape is guaranteed only when the gradient has a nonzero component along the bottom eigenvector; sustained acceptance drives σ_b toward σ_min and the Newton limit.Source: §5.2.1 · pp. 16–17, §5.2.3 · p. 17

5.2.2 The negative mode clamp

• The negative-mode clamp bounds one-step amplification and remains finite in fp32, trading per-step escape magnitude for compounded escape across accepted steps.Empirically, the clamp engages during early saddle-rich sweeps and disengages after trust adaptation settles.Source: §5.2.2 · p. 17

5.2.4 Free per-trial horizon ranking

• Because Ritz decompositions make candidate horizons nearly free, horizon ranking tests {h_b/2, h_b, 2h_b}, although increased Lanczos depth outranked horizon ranking in the FINER-scale sweep.On a small MLP control benchmark, the ranked variant selected shorter horizons 68% and longer horizons 29% of the time.Source: §5.2.4 · pp. 17–18

5.3 ARC-Block with φ1 subsolver algorithm

• Algorithm 2 preserves the per-block Gauss–Seidel structure and isolates the φ1 step rule by holding block partitions, ordering, routing, and equal-depth oracle accounting fixed.The implementation uses L = 25 at evaluated INR scale, with per-trial cost L + 2 Hessian–vector products plus one acceptance evaluation.Source: §5.3 · p. 18

6.1 Subproblem solver verification

• At indefinite-matrix degree L, Krylov relative model suboptimality falls from 7.8×10^-2 at L=5 to 4.3×10^-5 at L=25 and approximately 10^-15 at L=50.For positive-definite n=200 matrices, machine precision is reached already at L=5.Source: §6.1 · p. 19

6.2 ViSIR-Nano ablation (15,235 parameters)

• CubicKrylov gains 10.9 dB over the damped-Newton fallback at 0.66× the grad-equivalent cost, while fresh phase-B gradients prevent 6–8.5 dB plateaus.On 20 images, it reaches 32.8±1.4 dB, leads gradient-regularized Newton on 17/20 and lazy cubic Newton on 19/20, at 59% of their oracle budget and approximately 14× less wall-clock.Source: §6.2 · pp. 19–20, Table 3 · p. 20

6.3 The 91.4M parameter single image ViSIR Benchmark

• At 91.4M parameters, CubicKrylov reaches 51.65 dB in 148 seconds, exceeding Chebyshev-ON by 9.90 dB and damped-Newton large-block steps by 29.4 dB.The advantage comes from genuine curvature-aware cubic updates on large tensors where eigendecomposition-based curvature is infeasible.Source: §6.3 · pp. 20–21, Table 4 · p. 21
• 64.4 dB is reached with stabilized CubicKrylov versus 64.2 dB with Chebyshev recurrence, while global ARC stalls at 4.0 dB.These results use 150 block sweeps; CubicKrylov requires approximately 11 GB solver memory, whereas the recurrence uses 7.0–8.7 GB total peak.Source: §6.3 · p. 21

6.4 Landscape Fingerprints Along the Adam Trajectory

• Tuned Adam reaches 6.3×10^-14 MSE at lr=10^-4 in 76k steps, while coupled landscapes retain κAdam≈5.5×10^5–1.5×10^6 and saddle-dominated FINER stalls near 65 dB.The fingerprints distinguish axis-aligned cases where diagonal preconditioning suffices from coupled or saddle-dominated cases requiring curvature models or negative-curvature escape.Source: §6.4 · pp. 22–23, Table 6 · p. 23

6.5 Small-Scale Hard-Landscape Controls

• The φ1 step reaches terminal losses 1–5 orders below the cubic step on every small benchmark, including 4×10^-10 on the 35k SIREN after 40 sweeps.Its acceptance loop costs more oracle budget, while the negative-mode branch performs 353 block trials and escapes a rotated coupled saddle.Source: §6.5 · p. 24, Table 7 · p. 24

6.6 FINER 2D Image Fitting: The φ1 Step at Full Convergence

• 124.7 dB mean train PSNR is achieved by blockwise ARC at FINER convergence, versus 65.2 dB for tuned Adam and 75.1 dB for SOAP.The FINER initialization has 81% negative spectral mass, decaying to 19% along tuned Adam’s trajectory.Source: §6.6 · p. 24
• 90.65 dB: Extending Lanczos depth selects L = 163, but at matched L = 163 and step budget the cubic step reaches only 72.1 dB versus φ1’s 90.65 dB, isolating the step rule as the source of the margin.The response flattens beyond L ≈102, while per-cell wall clock grows from 393 s to 3.3 ks.Source: §6.6 · p. 25
• 63.53 ± 3.83 dB: The tuned 500-step ARC-φ1 setup leads cubic-step blockwise ARC at 61.78 ± 4.22 dB, tuned Adam at 54.34 ± 1.48 dB, CubicKrylov at 47.21 ± 9.04 dB, and DSO-v2r at 35.95 ± 2.18 dB.The result averages 20 images and 3 seeds across 60 cells, with approximately 400 seconds per cell.Source: §6.6 · p. 25
• ∼3× below: ARC-φ1’s high-band residual power is ∼3× below tuned Adam’s, while every frequency band improves together and the residual is band-uniform within 10% (hl ≈1.10).ARC-φ1 improves on cubic-step control by ∼1.6× per band, consistent with its +1.75 dB mean PSNR margin.Source: §6.6 · p. 25, Table 8 · p. 25
• 133.5 dB: ARC-φ1 reaches 133.5 dB at full convergence, versus tuned Adam’s 78.2 dB, a +55.3 dB gap; Figure 1 shows visually indistinguishable reconstructions and the difference in pixelwise error maps.ARC-φ1 runs 2,824 sweeps in 17,173 s and is still ascending, while Adam runs 1,000,001 steps in 4,214 s.Source: §6.6 · p. 25, Figure 1 · p. 26

• The proposed method uses per-tensor blocks, independent cubic constants and acceptance tests, and a Chebyshev-bounded Krylov subsolver driven only by Hessian–vector products.These design choices define the comparison with recent ARC-based approaches.Source: §7 · p. 27
• Table 10 compares related ARC-based families by curvature computation or approximation, largest demonstrated scale, and exactness of the cubic-model step on true curvature.Source: §7 · p. 27
• Table 11 compares the corresponding inner subproblem solvers by computation and working memory, while Appendix D reviews the broader ARC and cubic-regularization literature.The review includes Krylov, blockwise, partition-based, quasi-Newton, stochastic, lazy, subspace, multilevel, trust-region, and deep-learning-scale second-order methods.Source: §7 · p. 27

• FINER requires approximately 33× more gradient-equivalents than converged Adam, while tuned first-order methods can match accuracy faster on axis-aligned ill-conditioned landscapes.The evaluation is limited to coordinate-network regression with full-batch losses; stochastic mini-batch training at transformer-language-model scale was not tested.Source: §8 · p. 27
• Low-precision Hessian–vector oracles degrade both subsolver families, so mixed-precision deployment requires an fp32 oracle.Global-minimum guarantees are impossible on general non-convex objectives, and gradient-dominance assumptions do not hold for sinusoidal INR landscapes.Source: §8 · p. 27
• The φ1 subsolver has no proven optimality or O(ε^-3/2) rate; only a Cauchy-type decrease rule and generic second-order transfer are established.The escape argument requires a nonzero gradient component along the bottom eigenvector, and monotonicity follows from the outer per-block acceptance test.Source: §8 · pp. 27–28
• The full-convergence FINER result reaches 133.5 dB after 2,824 sweeps and 17,173 seconds, but uses one image and seed; broader testing remains necessary.The Adam-versus-ARC-φ1 comparison covers only two ESM images, with estimated ARC-φ1 costs of about 46 hours for 20 images and 253 hours for 110 images on one GPU.Source: §8 · p. 28
• Trajectory-dependent diagonal-dominance statistics do not ensure robustness: a diagonal φ1 variant fails on FINER’s saddle-rich landscape.The motivating diagonal statistics depend on the search trajectory.Source: §8 · p. 28

9 Conclusion

• The optimizer partitions parameters by tensor and assigns each block an independent cubic constant and acceptance test under a monotone stabilized outer scheme.Its cubic subproblem solver is interchangeable, supporting lazy exact steps on small blocks and Krylov-based steps on large tensors.Source: §9 · p. 28
• CubicKrylov and stabilized blockwise ARC set Lanczos degree from a Chebyshev second-kind bound driven by the cubic shift, whereas φ1 reuses the Lanczos build with a trust-parameter budget.CubicCheby-DSO uses an O(n_b)-memory Chebyshev recurrence whose residual test can stop before the precomputed degree, matching stored-basis solvers mainly when memory limits the solve.Source: §9 · p. 29
• At 91.4M parameters, stabilized per-block adaptation reaches 64.4 dB, versus 51.65 dB without stabilization and 4.0 dB for global single-subspace ARC.Per-block adaptive cubic regularization improves feasibility and performance of evaluated second-order optimizers at practical neural-network scales, regardless of subproblem solver.Source: §9 · p. 29
• A recurrence-based solver such as CubicCheby-DSO may offer greater advantages on architectures where the (L+1)-vector basis exceeds memory across more blocks.This potential follows from avoiding the stored-basis footprint that grows with Krylov dimension.Source: §9 · p. 29
• The landscape analysis attributes the gains to cross-coupled ill-conditioning and saddle structure, where coordinatewise or curvature-blind methods stall at relatively high-loss stationary points.The same measurements indicate settings in which the added second-order cost is wasteful.Source: §9 · p. 29
• The φ1 step combines Newton behavior on stiff modes, gradient behavior on flat modes, and clamped exponential escape on negative modes, improving hard-control accuracy by one to five orders of magnitude.On sinusoidal INR landscapes dominated by flat and saddle regions, φ1 is the stronger terminal-accuracy implementation than CubicKrylov, while its single-step optimality and tens-of-millions-parameter evaluation remain future work.Source: §9 · p. 29
• 133.5 dB: ARC-φ1 reaches this converged FINER accuracy versus Adam’s own converged 78.2 dB, a +55.3 dB advantage that SOAP and L-BFGS do not narrow.At equal wall-clock time, ARC-φ1 exceeds tuned Adam’s extended-budget ceiling by +17.4 dB while using the same per-trial oracle cost and memory envelope as CubicKrylov.Source: §9 · p. 29

A Feasibility of Second-Order Steps at the 91.4M-Parameter Scale

• The 91.4M-parameter model uses exact cubic steps on all 45 tensors: 23 small blocks use Algorithm 1 and 22 large blocks use matrix-free cubic-Krylov steps, with no first-order fallback.The largest decoder block contains 88,473,600 parameters, or 96.8% of the model.Source: A Feasibility of Second-Order Steps at… · p. 36
• The largest block requires 11 Hessian-vector products and approximately 3.9 GB for its Lanczos basis, while dense Hessian storage would require approximately 3 × 10^7 GB and factorization about 7 × 10^23 operations.The oracle charge is independent of block size, but HVP time and Lanczos-basis memory scale linearly with tensor size.Source: A Feasibility of Second-Order Steps at… · p. 36
• Global ARC stalls at 4.0 dB PSNR, whereas CubicKrylov reaches 51.7 dB; a block-σ variant reaches 61.9 dB after 13 sweeps, and stabilization raises its best to 64.4 dB.Global σ reaches 5×10^11 and rejects 46% of trials; per-block σ values span approximately 10^-5 to 10^7 across tensors.Source: A Feasibility of Second-Order Steps at… · p. 37

B The Cost Structure of the Subsolver: stored basis (CubicKrylov) vs Chebyshev recurrence as a subproblem solver

• Stored-basis Lanczos minimizes the cubic model over the Krylov subspace and reuses it across secular trials, whereas recurrence methods require fresh sweeps, multiplying HVPs despite lower memory and per-HVP overhead.The recurrence stores three vectors, while Lanczos stores L+1 vectors and performs reorthogonalization; one HVP dominates wall-clock cost at scale.Source: B The Cost Structure of the Subsolver:… · p. 38
• At equal nano budget, the best hybrid reaches 35.18 dB at 14.4 HVPs per block step, while stored-basis CubicKrylov reaches 36.64 dB; all hybrids use O(1) solver memory.The hybrids trail CubicKrylov by 1.5–3.1 dB because secular trials and rejection retries require fresh recurrence sweeps.Source: B The Cost Structure of the Subsolver:… · p. 39
• Refined hybrids improve nano trade-offs: v2r reaches 35.46 dB at 13.2 HVPs per block step, while grS reaches 34.93 dB at 12.8 HVPs but sacrifices 0.5 dB.Gradient-triggered spectral-bound refresh is the only refinement that improves both quality and cost over fixed EMA refresh.Source: B The Cost Structure of the Subsolver:… · p. 39
• On ESM-20, v2r averages 63.8 ± 4.0 dB with zero failures, exceeding CubicKrylov’s 58.5 dB; v3 averages 61.2 ± 4.0 dB, while grS fails 6/60 runs.Per-block ARC controls remain higher at 67.0–67.4 dB; successful grS runs average 60.4 ± 6.6 dB.Source: B The Cost Structure of the Subsolver:… · p. 40

B.1 Memory-Matched Comparison

• Memory-matched tests show 35.6 dB at Lmax = 2 and 36.2 dB at Lmax = 4 versus 36.4 dB for unbounded ARC, while recurrence runs use 7.0–8.7 GB versus approximately 11 GB.At equal wall clock, accuracy is non-monotone in degree, with Lmax = 10 performing best; measured peak memory is 9.6 GB across tested bounds.Source: §B.1 · p. 40

B.2 Robustness to a bf16 HVP Oracle

• The evaluation rebuilds loss, gradient, and HVP graphs under bfloat16 autocast, isolates subsolver numerics with fp32 accept-or-reject evaluations, and tests both solvers at nano and 91.4M scales.The basis solver additionally relies on mutual consistency across L successive HVP columns, whereas the recurrence uses each HVP once with fixed coefficients.Source: §B.2 · pp. 40–41
• At the tail, stationarity residuals reach 0.17 for the basis solver and remain 0.75–0.98 for the recurrence, while nano rejection rises to 27% for the basis solver.Noise enters the per-block ratio test through the model decrease and damages the returned steps despite preserved basis orthogonality.Source: §B.2 · p. 41
• bf16 preserves sampled basis orthogonality at fp64 roundoff, with max |QᵀQ − I| ≈10^-16–10^-13, so degradation enters through projected-model numerics.The recurrence’s fixed-coefficient solve is especially vulnerable, while the adaptive basis method can partially compensate for noisy products.Source: §B.2 · p. 41
• −4.5 dB versus −0.9 dB at 91.4M: the recurrence loses more than the adaptive basis solver under bf16 HVP noise.The recurrence applies each noisy Hessian–vector product once with coefficients fixed in advance, whereas the adaptive basis solve can partially compensate.Source: §B.2 · p. 41

Experimental setup

• Experiments cover SIREN, FINER, and real-valued Gabor WIRE on 240 × 240 RGB regression, plus Thai-statue SDF fitting with IGR supervision.Each image uses one three-hidden-layer, width-256 network with approximately 199k parameters; Adam and SOAP are tuned to their source papers, while ARC-block and CubicKrylov run to plateaus.Source: C Experiments on INR architectures · p. 41, Table 15 · p. 42

Residual analysis

• On SIREN, second-order runs reduce DC and Low residuals to approximately 10^-7–10^-6 per coefficient, 3–6 orders below Adam, while matching high-band accuracy.The resulting hl values are large because the denominator is near the precision floor; in absolute per-band residual power, ARC-block is at or below Adam in every band.Source: C Experiments on INR architectures · pp. 41–42

Results

• 120–129 dB on FINER, versus Adam’s 65.2 dB in 2D; in 3D, Adam’s chamfer error is 4.6–5.5× above every curvature-aware method.The blockwise cubic step continues improving with a band-uniform residual, while additional budget does not change Adam’s FINER results.Source: C Experiments on INR architectures · p. 42
• On SIREN, CubicKrylov reaches the best SDF test accuracy at 49.5 dB, but requires 8.3× Adam’s wall time.The methods are comparable at convergence, while the wall-clock comparison favors Adam.Source: C Experiments on INR architectures · p. 42

D Extended Comparison with ARC-Based Approaches

• The method reuses per-block Krylov bases with individual σ_b values and Hessian–vector products, unlike full-space or globally regularized ARC alternatives.Related approaches use reused polynomial or rational Krylov spaces, factorization-based extended Krylov bases, random projections, or one global regularizer.Source: D Extended Comparison with ARC-Based… · p. 42
• ARC-Block updates every parameter block with its own cubic constant and acceptance test, contrasting with methods that select one block or omit cubic regularization for partition subsets.Cristofari (2024) greedily selects one block per iteration, while Wolinski (2024) uses partition-induced projections and one learning rate per subset.Source: D Extended Comparison with ARC-Based… · p. 42
• The CubicKrylov and φ1 subsolvers apply the exact blockwise Hessian through Hessian–vector products, whereas quasi-Newton cubic methods replace it with low-rank curvature surrogates.Ranganath et al. (2025) and Forristal et al. (2022) provide closed-form or matrix-free cubic solves using limited-memory quasi-Newton structure.Source: D Extended Comparison with ARC-Based… · p. 43
• The framework couples recurrence stabilization with per-block adaptive cubic constants and an outer acceptance test, rather than analyzing recurrence gain only under adversarial Hessian perturbations.Its stability is demonstrated empirically at the 91.4M-parameter scale, while the cited recurrence analysis proves a closed-form bound for the recurrence in isolation.Source: D Extended Comparison with ARC-Based… · p. 43
• The study is limited to full-batch deterministic training: combining per-block acceptance with stochastic or variance-reduced oracles is left open.Related work supplies inexact cubic certificates, probabilistic acceptance conditions, and Hessian-reuse cost models, but none combines them with per-block adaptive cubic constants at network scale.Source: D Extended Comparison with ARC-Based… · p. 43
• Across broader second-order comparisons, the method partitions curvature by tensor and retains exact block Hessian actions, unlike level-wise, global, factored, trust-region, or non-cubic quadratic approaches.The cited alternatives include multilevel and randomized subspace methods, global trust-region or regularizer schemes, Gauss–Newton or Fisher factorizations, and Krylov solvers attached to quadratic models.Source: D Extended Comparison with ARC-Based… · pp. 43–44
• ARC has also been extended to Riemannian, constrained, minimax, and federated settings, but those works do not partition regularization within a single training problem.These cited extensions transfer the ARC template to other problem classes rather than adopting the paper’s per-block regularization structure.Source: D Extended Comparison with ARC-Based… · p. 44

E.1 Iso-time results

• The iso-time protocol grants every optimizer the same 148-second wall-clock budget, allowing cheaper first-order and diagonal methods to execute many more steps.The budget equals the 100-step time of CubicKrylov on the idle GPU.Source: §E.1 · p. 44

E.2 Convergence vs. oracle cost (gradient-equivalents)

• The gradient-equivalent metric is hardware-independent and credits methods that reuse curvature information, unlike step count or wall-clock time.PSNR is plotted against cumulative gevals in Table 17 and Figure 3.Source: §E.2 · p. 44
• CubicKrylov costs approximately 368 gevals per step, compared with approximately 8000 for full-Hessian Chebyshev-ON/OFF/Newton steps, a roughly 22× reduction.The full-Hessian methods rebuild exact small-block Hessians each step, while CubicKrylov uses lazy small-block Hessians and a degree-10 Krylov solve on large tensors.Source: §E.2 · p. 44
• Within CubicKrylov’s 100-step oracle budget of 36,836 gevals, it reaches 67 dB, whereas full-Hessian methods reach at most 19 dB after about five steps and first-order methods plateau near 20 dB.The oracle metric counts gradient and Hessian-vector evaluations in gradient-equivalents, with one HVP approximately equal to one backward pass.Source: §E.2 · p. 44

E.1 Iso-time results

• At T = 148 s, CubicKrylov leads widely, while Adam and SOAP plateau near 20–21 dB and Muon and AdaHessian perform lower despite thousands of steps.The comparison is on a single image reconstruction, so PSNR reflects reconstruction quality rather than generalization.Source: §E.1 · p. 44, Table 16 · p. 45, Figure 2 · p. 45

E.2 Convergence vs. oracle cost (gradient-equivalents)

• Block-size independence is crucial because the matrix-free Krylov step lowers curvature-aware cubic-step cost from approximately 8000 to approximately 368 gevals.This reduction is attributed to the Krylov step described in Section 4.Source: §E.2 · p. 45

Benchmark design

• The benchmark covers the first 20 ESM images with three seeds, ten methods, and 600 runs, giving each reference method twice CubicKrylov’s per-pair grad-equivalent budget.CubicKrylov first runs its 100-step ViSIR-nano experiment; fixed step counts are avoided because a dense Hessian build costs 15.2k grad-equivalents.Source: F Multi-Image, Multi-Seed ViSIR-Nano… · p. 45

Results

• 32.8±1.4 dB: CubicKrylov beats gradient regularized Newton on 17/20 images and lazy cubic Newton on 19/20, using 59% of their oracle budget and approximately 14× less wall-clock.The mean margins are +0.3 dB and +0.7 dB, with Wilcoxon p-values 3.9×10^-4 and 3.8×10^-6, respectively.Source: F Multi-Image, Multi-Seed ViSIR-Nano… · p. 45
• At nano scale, lazy m=n methods have overlapping quality distributions with CubicKrylov, so its advantage comes from cheaper matrix-free block steps rather than superior step quality.First kind spectral preconditioned gradient descent falls between the variants at 27.7±3.3 dB, while its rank-10 preconditioner cannot equalize SIREN contraction rates.Source: F Multi-Image, Multi-Seed ViSIR-Nano… · pp. 45–46

Reference-method analysis

• Every other configuration loses on all 20 images, as dense rebuilds exhaust the budget quickly and SSCN’s τ=64 subspace cannot match full-space curvature.DSO-BlockHess rebuilds all layers in about 21 steps or three steps depending on frequency; plain Newton still plateaus at 9.2 dB under an equal-time control, whereas abs-eig recovers to 33.1 dB without the budget rule.Source: F Multi-Image, Multi-Seed ViSIR-Nano… · p. 46

Run-to-run variability

• 43% coefficient of variation makes CubicKrylov the most variable method, with per-image means from 14.9 to 31.1 s and a worst single run of 60.4 s.Harder images trigger more rejected steps, fresh lazy Hessian rebuilds in phase A, and additional adaptive Mb retries in phase B.Source: G Wall-Clock Statistics of the… · p. 47, Table 19 · p. 47

Image dependence

• 2–6% variation characterizes fresh Hessian cubic Newton variants, the spectral preconditioner, and SSCN because dense-Hessian construction and eigendecomposition depend on block size rather than the fitted image.Lazy m=n methods vary moderately, with coefficient of variation around 9–12%, because timing is dominated by one dense build and 15,000 cheap iterations.Source: G Wall-Clock Statistics of the… · p. 47

DSO-BlockHess variability

• 5–17 s absolute runtimes characterize both DSO-BlockHess variants after early oracle-budget exhaustion; abs-eig has 2% coefficient of variation, versus 30% for plain Newton.The abs-eig implementation eigendecomposes every per-layer Hessian and preconditions with absolute eigenvalues.Source: G Wall-Clock Statistics of the… · p. 47

Wall-clock comparison

• ∼14× mean wall-clock advantage separates CubicKrylov from the lazy cubic Newton references; even its 60.4 s worst run is ∼4.5× faster than their 274.9 s fastest run.The comparison aggregates 20 images × 3 seeds, or 60 runs per method.Source: G Wall-Clock Statistics of the… · p. 47, Table 18 · p. 47

Stage (i): inside the benchmark budget

• 19.9 dB: within the benchmark budget, DSO-Adaptive reaches only 12.6, 16.6, and 19.9 dB for K=5, 10, and 20, respectively, versus 28 dB for the first-kind spectral preconditioner.Approximately 96% of parameters are routed through full block-Hessian mode, rapidly consuming the gradient-equivalent budget.Source: H Equal Budget Control at Nano Scale… · p. 49

Stage (ii): the plateau

• 30.0 dB: unconstrained DSO-Adaptive with K=5 converges after approximately 3,000 steps and 6.53×10^6 grad-equivalents, requiring 8,867 block rebuilds.Ten steps are skipped because eigvalsh fails to converge on a near-singular EMA Hessian.Source: H Equal Budget Control at Nano Scale… · p. 49

Rebuild accounting

• 33.1 dB: abs-eig DSO-BlockHess improves from 8.2 dB under the benchmark rule when granted greater budget, while DSO-Adaptive improves from 12.6 to 30.0 dB.The second-kind DSO family therefore exceeds first-kind spectral preconditioning by 2–5 dB at nano scale.Source: H Equal Budget Control at Nano Scale… · p. 49

Algorithmic failure

• 8.2 dB: plain Newton DSO-BlockHess remains trapped at saddles despite 6.5×10^6 grad-equivalents, while plain cubic Newton crashes at step 9 after Hessian eigendecomposition failure.The failures reflect algorithmic instability rather than insufficient budget: absolute-eigenvalue Newton lacks the second-kind saddle escape mechanism, and dense Hessian recomputation fails numerically.Source: H Equal Budget Control at Nano Scale… · p. 49

Stage (iii): interpretation

• 37.21 dB: global-Krylov ARC edges CubicKrylov at 36.64 dB in the equal-budget control, while per-block ARC variants reach 36.37–36.49 dB.CubicKrylov leads the budgeted benchmark methods at both 17.7k and 6.5×10^6 grad-equivalents, but global-Krylov ARC is the only Table 20 entry ahead of it.Source: H Equal Budget Control at Nano Scale… · p. 50, Table 20 · p. 50

ARC-global and the scale boundary

• 61.9 dB: the per-block variant remains effective on the 91.4M-parameter model, whereas identical global-Krylov ARC stalls at 4.0 dB after its global regularization weight rises to approximately 10^10.At nano scale, nearly all global ARC trial steps are accepted and σ reaches 10^-8; at full scale, one global σ must satisfy all directions simultaneously, causing escalating rejections.Source: H Equal Budget Control at Nano Scale… · p. 50