Making Analog Training Scale: Co-Designing Mapping, Optimizer, and Converters

1 INTRODUCTION

• Scaling is difficult because training errors accumulate across evolving iterations, while adaptive updates and widening inner products strain noisy devices and fixed converter ranges.Prior studies mostly evaluate iterative AIMC training at a few million parameters; tens-of-millions examples focus on fine-tuning or assume precise weight-state control.Source: §1 · pp. 1–2
• Digital weight-gradient formation, optimization, and residual accumulation retain precision-sensitive work in logical coordinates while transferring consolidated updates through threshold-triggered open-loop pulses.Forward and backward MVMs remain in analog arrays, providing an optimizer-agnostic logical update interface.Source: §1 · p. 3
• The framework couples logical-to-conductance mapping with dynamic converter-range alignment to preserve signal fidelity across model dimensions and provide finite-precision error bounds.The dynamic max-alignment policy is proved asymptotically optimal within the norm–rail converter family.Source: §1 · p. 3
• L ∝ N^-0.231 for analog training versus L ∝ N^-0.238 for digital AdamW across five model scales, despite a persistent validation-loss gap.The evaluation uses architecture-level simulations calibrated to experimental ECRAM measurements and includes hardware non-idealities.Source: §1 · p. 3

1.1 AIMC HARDWARE NON-IDEALITY MODEL

• The hardware model includes finite-resolution saturating converters, noisy crossbar outputs, and bounded, granular, state-dependent conductance updates.Signed logical weights are represented using reference-array subtraction and a logical-to-conductance scale.Source: §1.1 · pp. 3–4

1.2 RELATED WORK

• Prior approaches mitigate asymmetric updates or separate analog MVMs from digital weight-gradient formation; this work emphasizes optimizer, mapping, converter co-design and scalability.Analog mixed-precision training is described as separating low-precision analog MVMs from high-precision digital weight gradients.Source: §1.2 · p. 4

2 HOLISTIC CO-DESIGN FOR MIXED-SIGNAL AIMC SYSTEMS

• The pipeline computes forward and transposed backward MVMs in analog arrays, while W-grad computation and optimization remain digital; a digital residual buffer accumulates sub-threshold updates before open-loop writes.Analog rank updates could reduce update complexity from O(D²) to O(D), but raise device-endurance concerns.Source: §2 · pp. 4–5

2.1 WHY WEIGHT-GRADIENT ACCUMULATION SHOULD REMAIN DIGITAL?

• FP8 quantization of weight gradients catastrophically impairs convergence, unlike FP8 quantization of weights or activations; gradient accumulation error rises sharply with cancellation.The diagnostic quantizes intermediate error signals and parameter gradients while keeping reductions in FP32.Source: §2.1 · pp. 5–6, Figure 3 · p. 6
• Fragmenting a fixed-batch update across more quantized full-rank transfers increases one-step realized-update error through repeated write-path discrepancies.The controlled comparison fixes total batch size at 128 and does not establish end-to-end training failure.Source: §2.1 · p. 6

2.2 LOGICAL WEIGHT REPRESENTATION AND PHYSICAL MAPPING

• Both fixed-logical-range and native-AbsMax controls underperform the proposed fixed-scale mapping in final validation loss across model depths.The mapping decouples width-dependent logical initialization from width-independent physical conductance occupancy.Source: §2.2 · p. 7, Figure 4 · p. 7

2.3 CONVERTER ALIGNMENT BY DIGITAL RESCALING

• For sub-Weibull input tails, max-aligned distortion grows at most logarithmically with dimension, including O(log D) for Gaussian tails and O(log² D) for Laplace-type tails.The max-aligned rail is a low-overhead, distribution-agnostic approximation within the norm–rail family.Source: §2.3 · p. 8
• At D = 48, from four bits onward, max alignment is within 10% of calibrated distortion for both Gaussian and Laplacian inputs.The max-aligned curve approaches and at higher precision overlaps the calibrated optimum.Source: §2.3 · p. 8, Figure 5 · p. 8

Experimental setup

• The scaling evaluation compares matched digital AdamW training with analog training on lithium-ion and metal-oxide ECRAM tiles using the proposed recipe.Decoder-only Transformers were trained from scratch on OpenWebText following the Chinchilla scaling rule.Source: §3 · p. 9

Convergence and Scaling Laws

• L ∝ N^-0.231 for analog training, closely tracking L ∝ N^-0.238 for digital training, despite a persistent validation-loss gap at each evaluated scale.Models ranged from 6.8M to 123.6M parameters and were trained from scratch on OpenWebText under matched schedules.Source: §3 · p. 9

Bridging from baseline to This Work

• MpSGD fails across scales; MpAdam only slightly mitigates collapse, while converter alignment and tailored weight initialization yield stable convergence from Tiny through Medium.Converter alignment jointly manages noise and dynamic-range bounds.Source: §3 · p. 9

Conclusion

• The framework supports training decoder-only Transformers from scratch on OpenWebText up to 123.6M parameters in simulations calibrated to measured ECRAM characteristics.The authors conclude that compounded modeled nonidealities do not inherently preclude scaling AIMC training dynamics to larger models.Source: §4 · p. 9

Limitations

• The main scaling experiments use a single random seed, leaving run-to-run variability and uncertainty in fitted scaling trends unquantified.The simulated 16-bit DACs and 9-bit ADCs may incur substantial physical energy, area, and latency overheads, which the study does not model; results therefore establish feasibility, not verified hardware-efficiency advantage.Source: §4 · p. 9

Future Work

• Future work will align simulations more closely with measured devices and seek robust methods that preserve scaling behavior at substantially lower DAC/ADC precision.Source: §4 · p. 9

• Generative AI tools assisted with theoretical work, methods and experiments, analysis, coding, literature tasks, and drafting; authors reviewed the resulting work, and AI was not used to generate synthetic datasets or clean or reformat datasets.Source: AI USE STATEMENT · p. 10

• The architecture-level simulation uses no human participants or personal or sensitive data and releases no new dataset.The paper distinguishes simulated results from fabricated-hardware evidence and makes no safety, reliability, or societal-effect claims beyond the simulated setting.Source: ETHICS STATEMENT · p. 10

A SCALING LANDSCAPE OF ANALOG TRAINING

• The prior frontier included simulated vision models and fixed image datasets, while this work targets a 123.5M-weight decoder and billion-token language-model stream.The landscape also identifies gaps in physical evidence and integrated backward and optimizer paths for scalable on-chip training.Source: A SCALING LANDSCAPE OF ANALOG TRAINING · pp. 18–19

C SIMULATION DETAILS AND ADDITIONAL SIMULATIONS

• Transformer dense projections use analog crossbar MVMs, while attention-score computation, softmax, normalization, and other nonlinear operations remain digital.The appendix describes AIHWKIT simulations for experiments testing theoretical results on synthetic and real datasets.Source: C SIMULATION DETAILS AND ADDITIONAL… · pp. 20–21

C.1 PRECISION AND GRADIENT-ACCUMULATION DIAGNOSTICS (FIGURE 3)

• FP8 quantization of weights or activations caused almost no validation-loss degradation in the digital diagnostic, whereas FP8 gradient quantization impaired convergence.The accumulation replay isolates sequential rounding error for captured signals; it does not reproduce a hardware reduction tree or establish end-to-end training failure.Source: §C.1 · pp. 21–22

C.2 INITIALIZATION MAPS (FIGURE 4)

• The initialization comparisons include a matrix-specific native-AbsMax mapping and an illustrative PCM conductance normalization, rather than pooled Transformer histograms.The PCM illustration is distinct from the S-NW ablation, which changes read-scale interpretation while retaining S’s physical initialization.Source: §C.2 · pp. 22–23

C.3 INPUT-DAC RECONSTRUCTION-DISTORTION EXPERIMENT (FIGURE 5)

• At D = 48, AbsMax is within 10% of calibrated norm–rail distortion from four bits onward for both tested product laws.The inputs are independent product distributions, so this validates the distributional quantization calculation rather than representing every model layer.Source: §C.3 · pp. 23–24

C.4 ADDITIONAL ADC RANGE-ALIGNMENT DESIGNS

• Output-rail alignment reduces bound-management MVM retries, especially at larger widths, while retaining iterative bound management.The OUT condition changes non-head forward MVMs; the language-model head retains literal native S.Source: Figure 7 · p. 24, §C.4 · p. 24

C.5 SCALING ANALOG TRAINING (FIGURE 6 AND TABLE 3)

• Five decoder-only GPT models were trained from random initialization on OpenWebText using approximately 20N training tokens per model.Each update used 491,520 tokens, with matched AdamW coefficients and learning-rate schedules across the digital and analog paths.Source: §C.5 · pp. 24–25, Table 3 · p. 25
• The analog path used S-OUT with Li-ECRAM tiles, digital weight-gradient computation, and digitally retained optimizer moments and sub-threshold residuals.Analog runs used float32; attention-score products, softmax, normalization, and other nonlinear operations remained digital.Source: §C.5 · p. 25
• Loss was fit against compute, data, and parameter count using five completed endpoints per path, with dashed curves extrapolating to ten times the largest observed coordinate.The full-model compute proxy includes the vocabulary head and attention but excludes extra simulator work for dense one-hot embeddings.Source: §C.5 · p. 25
• Because compute, data, and parameter count covary across single-seed schedules, these fits cannot isolate separate parameter and data exponents or establish an asymptotic scaling law.Each condition was run once, so curve differences describe these runs rather than run-to-run variability.Source: §C.5 · p. 25

C.6 ZERO-SHOT DOWNSTREAM EVALUATION

• Answers were selected by mean token log-likelihood on task prompts, without gradient computation, calibration, or downstream fine-tuning.Tasks used fixed-seed samples of 128 examples when available, otherwise their full labelled split.Source: §C.6 · pp. 25–26
• Across tasks, digital and analog models show task- and scale-dependent macro-average screening accuracy, with individual-task variation in either direction.The comparison is descriptive, based on one realization per training condition and fixed evaluation samples.Source: §C.6 · p. 26, Table 4 · p. 26, Table 5 · p. 26
• The scaling-law pretraining losses remain lower for digital checkpoints, so the downstream screen does not remove the observed optimization gap.The evaluation is a controlled transfer screen, not a leaderboard evaluation or a reproduction of every benchmark’s canonical metric and prompt procedure.Source: §C.6 · p. 27

C.7 ALGORITHM ROBUSTNESS ACROSS GENERAL RPU DEVICE MODELS

• The comparison covers calibrated SoftBounds and eight measured-device models, with a matched Digital run as a non-analog reference.The presets are simulator fits to measured devices or arrays, and their trajectory visualizations retain cycle and device variation.Source: §C.7 · p. 27, Figure 8 · p. 28
• Across all nine RPU models and four depths, S attains lower final loss than SFB in the matched one-seed comparison.Tables 8 and 9 report final Shakespeare validation loss after 5,000 iterations for the methods across device models.Source: Table 8 · p. 28, Table 9 · p. 29, §C.7 · p. 29
• MO-ECRAM has roughly seven times Li-ECRAM’s nominal state count but larger specified cycle update variation; the fitted models do not establish a general technology ranking.For S, the ECRAM models have closely matched final losses; for SFB, MO-ECRAM is lower at every tested depth.Source: §C.7 · p. 29

C.8 ROBUSTNESS TO DEVICE NON-IDEALITIES

• Performance stayed near baseline over substantial device-variation ranges but deteriorated sharply at the largest variation; input noise degraded perplexity faster than output noise.These single-seed simulator results do not establish robustness for measured devices, correlated variations, other device models, or other workloads.Source: §C.8 · p. 29, Figure 9 · p. 30

C.9 ADC AND DAC QUANTIZATION SENSITIVITY

• ADC-grid loss ranged from 2.326 at 8-bit forward/backward to 3.527 at 4-bit forward/backward; DAC-grid loss ranged from 2.273 to 2.529.The DAC endpoints were 8-bit forward/9-bit backward and 5-bit forward/4-bit backward; this one-seed configuration was more sensitive to ADC quantization, especially in the forward pass.Source: §C.9 · p. 30, Figure 10 · p. 31

C.10 TIED TOKEN EMBEDDING AND LANGUAGE-HEAD MAPPING

• The analog tied-matrix mapping increased final validation loss by 0.0922 and late-training iteration time by 0.928 ms (8.1%) versus the digital-head control.Disabling final LayerNorm affine decreased loss by 0.0056 (2.4632 to 2.4576) while increasing iteration time by 0.301 ms (2.4%); the fully digital reference is not a tied-matrix-only ablation.Source: §C.10 · pp. 31–32

D AIMC COMPONENT ABLATION

• MpSGD’s matched screen uses O’s initialization and I/O with vanilla SGD, so comparison against O’s MpAdamW does not isolate digital weight-gradient accumulation alone.MpSGD accumulates high-precision SGD updates in a digital buffer and programs the array when the accumulated update crosses the device threshold.Source: D AIMC COMPONENT ABLATION · pp. 32–33
• Combining width-stable initialization and logical parameterization in S improves consistently with depth; initialization or logical parameterization alone yields only limited improvement.S keeps gradients, optimizer state, and transfer residuals in a common logical coordinate system, with increments transferred open-loop without explicit weight readback.Source: D AIMC COMPONENT ABLATION · pp. 33–34
• S-OUT improves on S, and S-PIO improves further, showing that read-path non-idealities remain after stabilizing the logical-update problem.S-OUT combines S’s update path with input-noise management, forward bound management, and dimension-aware matrix-specific ADC rails.Source: D AIMC COMPONENT ABLATION · pp. 33–34
• Explicit F/B periphery does not improve corresponding O- or S-based profiles at default precision; its effect depends on conductance scale, parameterization, and optimizer.Disabling iterative retries left S essentially unchanged with native input-noise management retained; deterministic replay observed no forward-ADC retry, which indicates operating-point headroom rather than zero clipping probability.Source: D AIMC COMPONENT ABLATION · pp. 33–34

E OPTIMIZER DEPENDENCE OF THE LOGICAL UPDATE

• The screen changes only the digital logical-weight update before the same quantized array transfer, with matched digital controls and fixed hyperparameters.It uses one seed and does not establish a universal optimizer ranking; the tested rules include matrix-oriented Muon, Moonlight, and Shampoo, plus Lion, Adafactor, and RMSprop.Source: E OPTIMIZER DEPENDENCE OF THE LOGICAL… · pp. 34–35
• Muon wins at every depth in Digital, Lion at every depth in S, and Lion at L = 2 versus Moonlight at L = 4, 6, and 8 in SFB.Shampoo, Adafactor, and RMSprop are consistently worse under the unchanged hyperparameters; SFB raises final loss for every matched optimizer–depth pair, though the size of the gap varies.Source: E OPTIMIZER DEPENDENCE OF THE LOGICAL… · pp. 35–36, Table 12 · p. 36

F.1 PERIODIC AVERAGING FOR NORM-NORMALIZED DIRECTIONS

• For varying norm orders and phases, the periodic rounding-error function averages to 1/12 as its frequency grows.The averaging holds uniformly in phase for directions on the open faces, including sequences of norm orders converging to infinity.Source: §F.1 · pp. 36–37

F.2 PROOF OF THEOREM 1

• For D ≥ 2, the proof obtains a common positive K^-2 leading constant for optimized distortion and max-rail distortion, yielding the theorem’s asymptotic ratio.The one-dimensional case has zero distortion for every K; the result relies on absolute continuity of the energy-weighted direction measure on open faces, while a joint density is sufficient but marginal absolute continuity alone is not.Source: §F.2 · pp. 37, 39
• The high-resolution threshold can depend on the distribution, even for smooth bounded product distributions, and a positive lower constant is not uniform over all dimensions and precisions.The proof notes examples where max-rail distortion becomes arbitrarily small near rail endpoints and where the maximum-moment expression grows with dimension.Source: §F.2 · p. 39

F.3 A QUANTITATIVE THRESHOLD UNDER BOUNDED VARIATION

• A conservative explicit precision threshold follows under a uniform bounded-variation bound on weighted direction densities, which controls their concentration across norm orders.The threshold may depend on distribution and dimension; absolute continuity alone does not provide the required variation bound.Source: §F.3 · pp. 39–40
• The inequalities remain valid when the finite quantizer clips coordinates, and clipping forces sufficient mean squared rail.Source: §F.3 · p. 41
• For ε ∈ (0, 1/3), K ≥ ⌈HD,T/(3ε)⌉ implies K ≥ HD,T and HD,T/K ≤ 3ε, yielding threshold (30) with explicit threshold (34).The derivation takes the infimum of the lower bound over p ∈ [1, ∞] and A > 0, then multiplies by cV.Source: §F.3 · p. 42

F.4 TAIL-DEPENDENT UPPER BOUNDS FOR A FIXED LAW: LONG VERSION AND PROOF OF THEOREM 2

• Theorem 7 gives tail-dependent bounds for arbitrary laws on R^D under stated coordinatewise assumptions, without requiring independent or identically distributed coordinates.The bounds cover bounded, sub-Weibull, Pareto-type, and finite-moment conditions.Source: §F.4 · p. 42
• If every coordinate is bounded in magnitude by B almost surely, then E[||x||∞^2] ≤ B^2.Source: §F.4 · p. 42
• Sub-Weibull coordinate tails yield E[||x||∞^2] ≤ L_s^2(log(2M_sD))^(2/α), including sub-Gaussian, sub-exponential, and heavier stretched-exponential tails.The stated classes correspond to α = 2, α = 1, and smaller α, respectively.Source: §F.4 · p. 42
• For Pareto-type coordinate tails with β > 2, E[||x||∞^2] ≤ [β/(β − 2)]L^2(MD)^(2/β), with D^(2/β) dependence sharp up to a slowly varying factor for iid regularly varying coordinates.The bound itself requires no coordinate independence.Source: §F.4 · p. 42
• Finite rth absolute moments for every coordinate, with r > 2, are another stated condition for a bound on E[||x||∞^2].Source: §F.4 · p. 42
• When the total second moment is infinite, a finite squared-error guarantee requires additional assumptions, truncation, or a lower-order distortion criterion.Source: §F.4 · p. 42
• The proof derives fixed-law distortion bounds by substitution into (19), using a union bound and tail integration for the remaining cases.For sub-Weibull tails, it splits the integral at L_s(log(2M_sD))^(1/α) and applies an incomplete-gamma tail bound.Source: §F.4 · p. 42

Remark 8 (Activation distributions in normalized Transformers)

• Pre-LayerNorm outputs have finite second moments for finite learned affine parameters, but residual, linear, or nonlinear operations can still produce activation outliers.The infinite-variance examples are described as boundary cases, not typical models of the normalized reads studied.Source: §F.4 · p. 43