PaperlayerView exact revision on arXiv

Source-linked AI summary

The Numerical Linear Algebra of Large Language Models

Abdelkader Baggag, Yousef Saad

arXiv:2610.04631v1cs.LGcs.NAmath.NA

TL;DR

The rapid development of AI raises concerns about risks and containment. The paper discusses NLA-relevant LLM methods, including low-rank adaptation, and surveys evidence that useful model behavior can occupy lower-dimensional subspaces than parameter counts suggest.

  • Problem

    Rapid AI development and its openness raise concerns about unprecedented risks and the need for containment.

    Source: §1.2 · pp. 4–5

  • Method

    The paper discusses NLA-relevant LLM findings and methods, including intrinsic dimension, low-rank adaptation, and advanced optimization.

    Source: §3.14.2 · p. 64, §4.2 · p. 77, §5 · p. 83

  • Results

    Empirical studies suggest language models work effectively in lower-dimensional subspaces; RoBERTa retains over 90% of its full performance in a randomly chosen 200-dimensional subspace.

    Source: §3.14.2 · p. 64

  • Takeaways & Limitations

    Low-rank adaptation offers a practically important approach to the time-consuming and impractical fine-tuning of pretrained models.

    Source: §4.2 · p. 77

    Fast-moving AI research means some topics covered in survey articles quickly become obsolete.

    Source: §6 · p. 92

Abstract

from arXiv · show

Numerical Linear Algebra (NLA) has consistently played a vital role in advancing science by providing tools to solve fundamental problems encountered in scientific and engineering applications. Over the decades, it has continually evolved to meet the demands driven by successive waves of scientific discovery. For instance, during the 1950s and 1960s, substantial efforts were devoted to developing methods for solving eigenvalue problems that emerged from the rapidly growing field of aerodynamics. This led to the discovery of the LR and QR algorithms. Later the attention turned to the solution of sparse linear systems that were common in applications like computational aerodynamics. Today we are experiencing yet another wave of major scientific advancement and NLA is once more at the heart of its development. This Machine Learning (ML) wave is proving to be utterly disruptive in science and engineering. Many tools in ML particularly Large Language Models (LLMs) are grounded in matrix and tensor methods. As we are approaching Artificial General Intelligence (AGI), it is clear that matrix methods will be called to play an even more significant role. For the numerical linear practitioner the speed of the current change makes it particularly challenging to adapt. This is a survey article that centers on machine learning techniques, with a particular focus on large language models. It has two main objectives. The first is to clarify the core concepts behind Large Language Models in a manner accessible to specialists in numerical methods. The second is to examine the key Numerical Linear Algebra concepts employed by LLM techniques, while also highlighting several significant recent contributions of NLA to the field.

1 Introduction and historical perspective

AI’s modern resurgence followed earlier cycles of excitement and retreat, as computing power and neural-network methods made increasingly complex models practical. The section traces the AI wave’s drivers and limits, then frames the article’s linear-algebra perspective and its scope.

1.1 Major factors of the advancement of AI

  • GPUs, deep neural networks, back-propagation, and accessible software tools helped revive AI and make complex multilayer models easier to train.Deep networks learn hierarchical features through many hidden layers, while TensorFlow and PyTorch helped make training codes accessible.Source: §1.1 · p. 3

1.2 The big AI wave and a comparison with chip manufacturing

  • 177M GPT-1 (2018) to 1.7T (estimate) GPT-4 (2023) anchors a four-model trend doubling every ≈4.6 months, which the authors say cannot persist.Model size is linked to a limited data corpus and may not rise much from current levels.Source: §1.2 · p. 4, Figure 2 · p. 5
  • AI is more open to contributions from varied sources than protected, highly competitive chipmaking, a difference the authors connect to both research benefits and misuse concerns.The text frames the contrast through the ‘bazaar’ and ‘Cathedral’ models.Source: §1.2 · p. 5
  • Aschenbrenner argued that another six OOMs of gains from 2023 to 2027 could bring a preschooler-to-high-schooler-sized qualitative jump; Figure 3 is schematic.The figure combines effective compute, algorithmic efficiency, and “unhobbling” factors as a scaling argument, not a precise prediction.Source: §1.2 · p. 5, Figure 3 · p. 6

1.3 Main ingredients of Deep-Learning

  • Modern deep learning draws on hardware, data, optimization, statistics, and NLA, while software packages support parallel processing across multiple GPUs.Transformer attention scales quadratically with token count, motivating hardware and algorithmic efforts toward linear or near-linear scaling.Source: §1.3 · pp. 6–7, Figure 4 · p. 7

1.4 Linear Algebra for AI

  • NLA’s matrix and tensor tools shape AI training and inference, while the article urges linear-algebra researchers to engage with deep learning despite fast-paced cultural differences.The article’s stated aim is an exposition of LLM ingredients from a matrix-theory perspective, not a complete account of the field.Source: §1.4 · p. 8

2 Deep Neural Networks

This section surveys deep neural networks as data-trained parameterized functions, covering MLPs, loss and theoretical foundations, and stochastic and adaptive optimization methods and their trade-offs. It also explains back-propagation as gradient propagation through computational graphs and layers, and highlights the role of matrix and tensor computations and attention.

2.1 Example of Multi-Layer Perceptrons

  • An MLP approximates a function by composing learned affine transformations and nonlinear activations across layers, with output dimensions and parameters determined by layer widths.The digit-classification example treats training as fitting parameters to labeled samples; activations include ReLU, sigmoid, tanh, Swish, GELU, and learned-gate SwiGLU.Source: §2.1 · p. 11

2.2 The loss function for MLPs

  • For classification, outputs are converted by softmax into probability distributions and trained with cross-entropy against one-hot labels; backward differentiation is known in machine learning as backpropagation.The section formulates the network over sample matrices, with each row representing a sample and internal layer outputs processed in blocks.Source: §2.2 · pp. 12–14

2.3 The Universal representation theorem

  • Cybenko’s universal representation theorem shows that continuous discriminatory activations suffice for single-hidden-layer functions to be dense in continuous functions on [0, 1]^n under the uniform norm.Sigmoidal functions are identified as an important discriminatory class.Source: §2.3 · p. 14

2.4 Stochastic Optimization techniques for DNNs

  • Nonconvex objectives, poor generalization from overfitting, and huge parameter counts make neural-network training challenging, motivating stochastic optimization [79].A low training cost does not ensure low error on unseen data; pursuing high accuracy with traditional optimization can worsen generalization.Source: §2.4 · p. 15

2.4.1 Stochastic Gradient Descent (SGD)

  • SGD replaces the costly full gradient of a finite-sum objective with a randomly sampled component gradient, while mini-batching averages several sampled functions to improve on slow single-sample updates.The learning rate may be fixed or adaptive; practical mini-batching often samples without replacement, although theory commonly assumes sampling with replacement.Source: §2.4.1 · pp. 15–16
  • Mini-batch SGD updates weights using gradients over batches, making computation manageable; randomized mini-batches can reduce sweeps to convergence and help avoid local minima and overfitting.One sweep through all batches is an epoch, and complex models may require tens of thousands of epochs to converge.Source: §2.4.1 · pp. 16–17

2.4.2 Adagrad and RMSprop

  • Adagrad adapts each parameter’s learning rate using past squared gradients, but its rate can vanish; RMSprop counters this exhaustion with an exponential moving average.Adagrad’s accumulated squared gradients enlarge the denominator over time, whereas RMSprop uses a running average instead of a sum.Source: §2.4.2 · p. 17

2.4.3 Momentum

  • Momentum modifies each SGD iterate by adding a multiple of the previous increment; setting µ = 0 recovers the original SGD scheme.The method is related to second-order acceleration, and the previous-step contribution is termed velocity.Source: §2.4.3 · p. 18

2.4.4 Adam and AdamW

  • AdamW yields better generalization than standard Adam and is often the default optimizer in large language models.Adam combines momentum with RMSprop using exponentially attenuated terms for the gradient and its variance; AdamW modifies weight decay for adaptive gradients.Source: §2.4.4 · pp. 18–19

2.5 Challenges of Deep Learning and the issue of generalization

  • A lower training loss does not ensure better inference: models can fit randomly relabeled data perfectly, while stochastic optimization helps find minima that generalize.The section frames generalization as a central challenge for non-convex, over-parameterized objectives with many possible solutions.Source: §2.5 · p. 20
  • Adaptive methods can converge faster yet generalize worse, while optimization methods’ memory demands favor simple optimizers such as SGD or Adam in large models.On CIFAR10, adaptive methods converged faster but generalized more poorly; models such as GPT3 (175B parameters) and Llama3 (405B) illustrate the memory challenge.Source: §2.5 · pp. 19–20

2.6 Computational graphs and back-propagation

  • Back-propagation computes derivatives by applying the chain rule backward through a computational graph, traversing nodes in topological order from the loss toward inputs and parameters.Forward evaluation first computes intermediate quantities; reverse-mode differentiation then propagates derivatives backward, summing contributions through the graph.Source: §2.6 · pp. 21, 22, 24
  • Back-propagation applies the chain rule through layer Jacobians to compute gradients with respect to biases and weights.For a layer’s weight matrix, the derivative with respect to each column is the same and equals σ(s_{l−1})^T.Source: §2.6 · pp. 24–25

2.7 Tensor Computations in DNNs

  • DNN computations use tensors to represent batched image data and preserve spatial or sequence structure, flattening CNN outputs only before a final dense layer.CNNs apply filters and pooling to spatial tensor dimensions, while language-model sequences retain the shape (batch_size, sequence_length, embedding_dim).Source: §2.7 · p. 25

2.8 NA thinking vs. ML/AI thinking: Attention

  • With abundant data, Nadaraya–Watson attention can approximate an unknown oscillatory function better than a cubic fit, using query-relevance weights to average observed values [101, 155].The weights are based on similarity between a query and the training keys; the kernel weights are positive and sum to one.Source: §2.8 · pp. 26–27

3 Large Language Models and Transformer Architecture

This section frames Transformers and large language models through linear algebra, explaining autoregressive prediction, token representations, and the role of position in sequence modeling.

3 Large Language Models and Transformer Architecture

  • Attention-based kernel regression captures the periodic target more accurately than a simple cubic approximation when the function’s form is unknown.The figure illustrates how a similarity-based weighted average can recover useful structure directly from data.Source: Figure 14 · p. 28

3.1 Language modeling and the rise of LLMs

  • An autoregressive language model estimates each next-token distribution from preceding tokens, and generation iterates this prediction step by step.The objective is cross-entropy, equivalently the training sequence’s negative log-likelihood; the objective itself did not fundamentally change with LLMs.Source: §3.1 · pp. 28–30
  • LLMs are large-scale autoregressive Transformer language models, with decoder-only Transformers trained on data and computational resources at scale.Training evaluates losses at positions in parallel under causal masking, while inference generates tokens sequentially.Source: §3.1 · pp. 29–30

3.2 Tokens, embeddings, and positional structure

  • A learned embedding maps symbolic one-hot tokens into dense vectors, while positional structure is needed because token embeddings alone do not encode order.The embedding matrix selects learned token vectors; positional information modifies the initial representation before attention.Source: §3.2 · pp. 30–31
  • Sinusoidal encodings add position to initial token representations, whereas relative biases and rotary embeddings alter attention using positional relations or position-dependent rotations.These mechanisms break permutation symmetry so attention depends on sequence order.Source: §3.2 · pp. 31–32

3.3 The Transformer architecture

The Transformer is presented as a stack of matrix-valued sequence operators. Its blocks combine attention and MLP updates, and decoder-only variants use causal masking for autoregressive language modeling.

3.3 The Transformer architecture

  • The section develops Transformer blocks as repeated combinations of self-attention, feed-forward transformations, normalization, and residual addition.It also describes the Pre-LayerNorm formulation and mapping of hidden states to vocabulary scores.Source: §3.3 · p. 33

3.3.1 Transformer models as stacked sequence operators

  • A Transformer repeatedly transforms a matrix of token representations through depth, replacing recurrent evolution of a single hidden state with composed layer maps.Each block preserves the sequence-length and feature dimensions of the representation matrix.Source: §3.3.1 · pp. 33–34

3.3.2 Pre-LayerNorm Transformer block

  • Each Pre-LayerNorm block applies attention and an MLP with normalization, adding both sublayer outputs into the same residual state matrix.Attention mixes information across token positions; the MLP mixes feature coordinates within each token.Source: §3.3.1 · p. 34, §3.3.2 · p. 34

3.3.3 Final normalization and logits

  • After the final block, layer normalization and the output un-embedding project hidden states to vocabulary logits, which row-wise softmax converts to probabilities.Each logits row contains unnormalized scores for vocabulary items at one token position.Source: §3.3.3 · p. 34

3.3.4 Encoder-only, decoder-only, and encoder-decoder variants

  • Encoder-only models use bidirectional self-attention, encoder-decoder models combine source and target stacks, and decoder-only models use masked self-attention [34, 113, 144].The decoder-only form dominates LLMs and is trained for autoregressive next-token prediction.Source: §3.3.4 · p. 35

3.4 Single-head self-attention

Single-head self-attention forms token interactions from learned projections, normalizes their scores into weights, and uses those weights to mix value vectors. Causal masking restricts decoder attention to current and earlier tokens.

3.4 Single-head self-attention

  • Single-head self-attention projects token representations into queries, keys, and values, computes pairwise similarity scores, and uses attention weights to mix sequence information.The mechanism includes a masked form for autoregressive language modeling.Source: §3.4 · p. 35

3.4.1 Query, key, and value projections

  • Learned query, key, and value projections provide the inputs for the score computation and the value content transported by attention.The value projection determines feature content after interaction weights are computed.Source: §3.4.2 · p. 37, §3.4.1 · p. 37

3.4.2 Scaled dot-product attention

  • Scaled query-key dot products form attention scores, with the factor 1/√dk stabilizing their magnitude before row-wise softmax.Each row of the resulting attention matrix is a probability distribution over token positions.Source: §3.4.2 · pp. 37–38

3.4.3 Attention output as weighted averaging

  • Each output token is a weighted average of value vectors, making the attention map linear in values while its coefficients depend nonlinearly on query-key scores.The softmax applied to projected-input inner products supplies the nonlinearity.Source: §3.4.3 · pp. 38–39

3.4.4 Masked self-attention

  • Causal masking sets forbidden future-token scores to −∞, yielding lower-triangular attention support so position i depends only on positions 1 through i [23].Exponentiation maps masked entries to zero before row-wise softmax.Source: §3.4.4 · p. 39

3.4.5 Rank and computational bottleneck

  • The pre-softmax score matrix has rank constrained by projection dimension dk, while its n × n size incurs O(n^2) arithmetic and memory costs in sequence length.This quadratic cost is a central bottleneck motivating efficient-attention variants.Source: §3.4.5 · p. 40

3.5 Multi-head attention

Multi-head attention computes distinct attention operations in learned projected subspaces and recombines their outputs. This structure can represent multiple relations, though the account notes both potential expressivity and redundancy.

3.5 Multi-head attention

  • Multi-head attention addresses the limits of one token-mixing map by computing several learned attention operations in parallel and recombining their outputs.This allows different token relations to be represented at once.Source: §3.5 · p. 40

3.5.1 Headwise projections

  • Each head uses its own query, key, and value projections, creating separate score, attention, and value pathways.In standard implementations, full-layer dv and dk often equal dmodel, with each head operating on a lower-dimensional slice.Source: §3.5.1 · pp. 40–41

3.5.2 Concatenation and output projection

  • Concatenated head outputs are mapped back into the shared model space by an output projection, equivalently summing each head’s projected contribution.The output projection returns the concatenated representation to the residual-stream space.Source: §3.5.2 · p. 41

3.5.3 Multi-head attention as a block-structured operator

  • From a linear-algebra perspective, multi-head attention is a sum of headwise operators, each with its own projected query, key, and value spaces and output map.Each head’s token-mixing operator acts on its projected value space before mapping back to the common residual stream.Source: §3.5.3 · pp. 41–42

3.5.4 Subspace structure, expressivity, and specialization

  • Separate learned subspaces let heads represent different token relations, potentially increasing expressivity, while some heads may be redundant or overlapping in function.The account describes specialization and partial redundancy empirically, while noting that the reason multi-head attention helps is not fully answered.Source: §3.5.4 · p. 42

3.6 Positional structure and sequence order

Self-attention alone does not encode token order, so positional structure changes the attention computation to make it sequence-aware. Decoder-only LLMs combine positional structure with causal masking, which restricts visible positions.

3.6.1 Why self-attention alone ignores order

  • Self-attention is permutation-equivariant: permuting token representations permutes its output, so position must be added for sequence order to affect computation.Token identity alone does not determine sequence structure.Source: §3.6.1 · p. 43

3.6.2 Absolute and relative positional mechanisms

  • Absolute encoding adds position to token embeddings before attention; the original Transformer uses sinusoidal vectors that represent token positions.This changes the geometry of the projected token similarities without changing the attention mechanism's form.Source: §3.6.2 · p. 43
  • Relative methods adjust attention scores using position offsets, while RoPE rotates queries and keys so their similarities reflect relative displacement.These mechanisms alter different stages of the attention pipeline.Source: §3.6.2 · p. 44

3.6.3 Sequence order in decoder-only LLMs

  • Decoder-only LLMs use causal masking to forbid attention to future tokens and positional structure to represent the visible positions relative to one another.Causal masking alone does not specify how earlier visible positions are arranged.Source: §3.6.3 · pp. 44–45

3.7 The MLP / feed-forward sublayer

The Transformer MLP mixes features within each token, complementing attention's mixing across token positions. Its expansion, nonlinear transformation, and compression create input-dependent feature maps; gated variants further alter this computation.

3.7.1 The feed-forward stage: expansion, compression, and channel mixing

  • The MLP expands each token representation from dmodel to dmlp, applies a nonlinearity, then projects it back to dmodel.Many implementations use dmlp ≈ 4dmodel.Source: §3.7.1 · p. 45
  • Unlike attention's mixing across token positions, the MLP acts independently on each token and mixes its feature channels.Attention uses an n × n matrix, whereas the MLP transforms each row separately.Source: §3.7.1 · p. 45, §3.7.4 · p. 47

3.7.2 Nonlinearity and input-dependent Jacobians

  • Nonlinear activations make the MLP an input-dependent family of affine maps, with ReLU activation patterns selecting distinct piecewise-affine regions.The resulting effective linear map depends on the input.Source: §3.7.2 · p. 46

3.7.3 Gated variants: SwiGLU and related designs

  • Llama-style MLPs use a gated three-matrix design such as SwiGLU, combining two projected branches elementwise before the down projection [129].The elementwise product introduces multiplicative gating into the feature transformation.Source: §3.7.3 · p. 46

3.7.4 Why the MLP dominates parameter count and compute

  • MLP stages contain most parameters and much of the compute in large dense Transformers, motivating sparse expert designs that typically replace the MLP rather than attention.The MLP accounts for roughly two-thirds of GPT-3 175B parameters, and a larger fraction in Llama-3 405B [36].Source: §3.7.4 · p. 47

3.8 Layer normalization and residual connections

Residual connections and normalization work together to support stable signal propagation through deep Transformer stacks. The section describes row-wise normalization, RMSNorm, and how normalization placement changes the residual path.

3.8 Layer normalization and residual connections

  • Residual connections carry an identity pathway through depth, while LayerNorm regulates the scale and centering of hidden representations to support optimization stability [15].LayerNorm acts on each token's feature coordinates rather than mixing token positions.Source: §3.8 · p. 47

3.8.1 RMSNorm as a modern variant

  • RMSNorm omits LayerNorm's mean subtraction and rescales by the input's root-mean-square magnitude, while retaining learned gain parameters.It is used in the Llama family and serves the same stabilizing architectural role as LayerNorm.Source: §3.8.1 · p. 48

3.9 Pre-LayerNorm, Post-LayerNorm, and the residual stream

  • Pre-LayerNorm applies normalization before each sublayer and preserves an exact identity residual path; Post-LayerNorm normalizes after residual addition.Modern LLMs typically favor Pre-LayerNorm because it improves optimization stability at depth.Source: §3.9 · p. 49, Figure 21 · p. 50, Figure 22 · p. 51

3.10 The Transformer block as a composite operator

A Transformer block composes structured operations that mix information across tokens and features, with residual additions writing updates into a shared stream. In decoder-only LLMs, causally masked blocks refine contextual representations across depth.

3.10.1 One block as a composition of structured maps

  • A Transformer block composes row-wise normalization, token-mixing attention, residual addition, a feature-mixing MLP, and a second residual addition.The block is a structured operator, not one undifferentiated nonlinear map.Source: §3.10 · p. 51, §3.10.1 · p. 52

3.10.2 What is linear and what is nonlinear?

  • Attention projections are linear, their score matrix is bilinear, and row-wise softmax makes the attention matrix nonlinear; the MLP and LayerNorm are nonlinear overall.The MLP is linear before and after its activation, while LayerNorm's centering and scaling depend on the input row.Source: §3.10.2 · p. 52

3.10.3 Residual addition and operator ordering

  • Attention followed by MLP generally differs from the reverse order because they mix token positions and feature coordinates, respectively.Residual additions and normalization placement also make the block's behavior depend on operator order.Source: §3.10.3 · p. 52

3.10.4 Progressive representation refinement

  • Repeated attention and MLP updates progressively refine token embeddings into contextual representations that support next-token prediction and other emergent capabilities.Shallow-layer representations remain closer to token-plus-position embeddings than deeper-layer representations.Source: §3.10.4 · pp. 52–53

3.11 Internal workings of decoder-only LLMs

  • Decoder-only LLMs stack causally masked Transformer blocks so each token's hidden state represents its visible prefix, then project final states to vocabulary logits and probabilities.The block equations remain locally the same, but attention is constrained by causality.Source: §3.11 · pp. 53–54

3.11.1 The residual stream as working memory

  • The residual stream accumulates attention and MLP contributions, allowing information gathered at one depth to be preserved and reused by later layers.Attention gathers from earlier positions; residual updates retain that information for further computation.Source: §3.11.1 · p. 54

3.11.2 Decoder-only LLM examples: GPT-3, Llama-3, and Gemma

  • GPT-3 175B and Llama-3 405B share the decoder-only template but differ in depth, width, feed-forward design, and attention configuration; Gemma emphasizes deployability and long-context engineering [50].GPT-3 uses a standard dense MLP, whereas Llama-style models use gated activations such as SwiGLU and grouped-query attention variants.Source: §3.11.2 · pp. 54–55

3.11.3 Parameter counting as a worked example

  • The parameter breakdowns show that MLPs exceed multi-head attention in GPT-3 and are even more dominant in Llama-3's gated design.The examples report totals of 175,191,908,352 parameters for GPT-3 175B and 405,872,984,064 for Llama-3 405B.Source: §3.11.3 · p. 55, Table 3 · p. 56, Table 4 · p. 56

3.12 Mixture-of-Experts Transformers

Mixture-of-Experts Transformers replace the shared dense feed-forward block with routed expert maps, making channel mixing sparse and input-dependent. This lets models increase total parameter capacity while keeping per-token computation closer to that of a dense model when few experts are active.

3.12.1 From dense feed-forward blocks to experts

  • MoE replaces the shared dense feed-forward block with expert maps and routing, so each token activates only a small subset of the experts.The standard dense block applies the same map to every token; the routed design can support a much larger total parameter count than the number activated per token.Source: §3.12.1 · pp. 56–57

3.12.2 Routing, sparse expert computation, and operator interpretation

  • A router scores experts and selects the top-k for each token; their outputs are combined using the corresponding routing weights.Typically k is much smaller than the total number of experts M, making channel-mixing computation sparse.Source: §3.12.2 · p. 58
  • Increasing the expert count raises model capacity while keeping active computation closer to a dense model when only a few experts are selected per token.The total expert pool scales with the number of experts, whereas the active parameter budget depends on the smaller number selected.Source: §3.12.2 · p. 58
  • Routing can overload a few experts and leave others rarely used, so MoE designs use load-balancing losses or capacity constraints.The design must manage both representational power and the distribution of computation across experts.Source: §3.12.2 · p. 58
  • Switch Transformer uses top-1 routing, Mixtral 8 × 7B routes each token to 2 of 8 experts, and gpt-oss provides a GPT-style MoE example.MoE architectures vary in expert counts, active experts, routing, balancing, and model scale.Source: §3.12.2 · pp. 58–59
  • MoE makes channel mixing input-dependent, extending the data-dependent computation already provided by attention into the feed-forward stage.Each token sees a sparse combination of expert operators selected according to its current state.Source: §3.12.2 · p. 59

3.13 Training dynamics

The section explains how autoregressive language models train through masked parallel operations and how later fine-tuning stages shape chat-oriented models. It also contrasts this matrix-based computation with recurrent training.

3.13.1 Autoregressive training, masking, and optimization

  • Autoregressive training uses cross-entropy, or sequence negative log-likelihood, to predict next tokens from the visible prefix.The model produces vocabulary distributions at each visible sequence position, attaching the training signal across positions.Source: §3.13 · p. 59, §3.13.1 · p. 60
  • Causal masking lets Transformers process known training sequences in parallel while preventing each position from seeing future tokens.At inference, generation proceeds one token at a time as each predicted token is appended to the context.Source: §3.13.1 · p. 60
  • Pre-LayerNorm Transformers train more stably than Post-LayerNorm variants in deep LLM regimes, with the skip path providing gradients a direct route across depth.The residual identity map is given as the reason for this stability difference.Source: §3.13.1 · pp. 60–61

3.13.2 Fine-tuning, alignment, and comparison with recurrent training

  • Chat-oriented models commonly undergo supervised fine-tuning on instruction-response examples and preference alignment using demonstrations, a reward model, and RLHF.These stages change the training distribution and, for preference-based stages, the optimization objective, while leaving the Transformer block map unchanged.Source: §3.13.2 · p. 61
  • Transformers enforce temporal dependence with causal masking and matrix operations, unlike recurrent models, which evolve hidden states through time and backpropagate through temporal chains.The section links masked matrix computation to alignment with modern parallel hardware.Source: §3.13.2 · p. 61

3.14 High-dimensional geometry, JL, and intrinsic dimension

The section uses high-dimensional geometry to explain how LLMs can represent many partially distinct features, and reviews evidence that useful representations and updates often occupy lower-dimensional subspaces. It also describes how representation dimension changes across model depth.

3.14.1 Johnson–Lindenstrauss, and representation geometry

  • Allowing bounded distortion ε permits encoding on the order of exp(ε · d) approximately orthogonal directions in d dimensions, increasing representational capacity beyond d perfectly orthogonal directions.The section uses the Johnson–Lindenstrauss perspective as geometric intuition for weak interference, not mainly as a dimensionality-reduction result.Source: §3.14.1 · pp. 62–63

3.14.2 Intrinsic dimension in language models

  • For RoBERTa, optimizing in a randomly chosen 200-dimensional subspace recovers over 90% of full fine-tuning performance, despite the much larger parameter dimension.The section presents this as evidence that useful fine-tuning updates can occupy low-dimensional subspaces.Source: §3.14.2 · pp. 64, 66
  • The section reports over 98% redundancy in LLM token embeddings and links low-dimensional structure to compression, distillation, pruning, and parameter-efficient fine-tuning.It also describes LoRa’s effectiveness as aligned with intrinsic subspaces, allowing adaptation with minimal parameter updates.Source: §3.14.2 · p. 65
  • BERT sentence embeddings can be linearly recovered from 384 of 768 dimensions with over 90% accuracy, suggesting lower-dimensional structure in the embeddings.The section defines intrinsic dimension through the smallest reduced dimension meeting a prescribed reconstruction or task-performance threshold.Source: §3.14.2 · p. 66
  • An illustrative gradient-energy example exceeds 90% cumulative energy within roughly 10 components, while reported effective gradient ranks in LLMs typically range from 100 to 1,000.The section describes these gradients as numerically low-rank, even for models with billions of parameters.Source: §3.14.2 · p. 67
  • Across reported studies, token-representation intrinsic dimension rises in early layers, peaks in intermediate layers, and declines toward the output.The section relates this expansion-compression pattern to diffusion into a broad latent space followed by compression onto task-relevant submanifolds.Source: §3.14.2 · p. 67

3.15 Interpretability through linear algebra

This section treats interpretability as analysis of Transformer operators and representations, not only input-output behavior. It examines attention transport, additive residual contributions, vocabulary-space probes, and partially disentangled heads and features, while noting limits to semantic interpretation.

3.15.1 Attention, residual streams, and logit-space probes

  • Attention matrices are row-stochastic token-to-token transport operators, but they are not guaranteed to explain the model’s predictions.They show which token positions are mixed, not why a particular output is predicted.Source: §3.15.1 · p. 68
  • Pre-LayerNorm residual states accumulate attention and MLP updates additively, enabling analysis of component contributions in their shared state space.The decomposition may be algebraically useful without having a uniquely meaningful semantic interpretation.Source: §3.15.1 · pp. 68–69, Figure 25 · p. 69
  • The logit lens projects intermediate residual states into vocabulary space, while the tuned lens adds a learned affine map before unembedding.Both use the vocabulary basis to compare intermediate representations across layers.Source: §3.15.1 · pp. 69–70

3.15.2 Head specialization, feature directions, and the limits of linear interpretability

  • Some attention heads implement recognizable operations such as copying, induction, or delimiter matching, but specialization is partial rather than a clean partition of labor.Induction-head analyses describe a copy-and-continue mechanism for in-context pattern continuation.Source: §3.15.2 · p. 70
  • Superposition means features may occupy overlapping residual-stream directions, so interpretation may require useful subspaces or stable directions rather than one axis per concept.Features are not guaranteed to be orthogonal or uniquely identifiable.Source: §3.15.2 · p. 70
  • Linear-algebra probes organize internal computations, but their semantic or causal meaning remains open, and early-layer lens projections may distort representations.Attention maps need not faithfully explain predictions, and residual decompositions can be semantically ambiguous.Source: §3.15.2 · p. 70

3.16 Scaling laws and emergent phenomena

This section surveys predictable scaling trends and threshold-like emergent behavior in large language models. It describes compute-optimal allocation of model size and data, while noting that the mathematical mechanisms behind scaling and emergence remain unsettled.

3.16.1 Empirical scaling laws and compute-optimal training

  • Language-model test loss follows approximate power-law trends with model size, data size, and compute, with diminishing returns represented by scaling exponents.The scaling-law schematic depicts test loss decreasing as N, D, and C increase.Source: §3.16.1 · p. 71, Figure 26 · p. 72
  • Compute-optimal scaling finds that balanced parameter and token allocations outperform single-axis optimization, motivating co-design of parameters, data, and compute.This guidance replaces crude trial-and-error scaling with a more disciplined resource-allocation picture.Source: §3.16.1 · pp. 71–72, Figure 26 · p. 72
  • GPT-3 exemplifies scale accompanying qualitative task behavior without a radically new architecture relative to GPT-2.The scaling hypothesis holds that performance can improve reliably by increasing model and data size without fundamental architectural changes.Source: §3.16.1 · p. 72

3.16.2 Emergent behavior, mathematical questions, and cautious interpretation

  • Emergent abilities are threshold-like capability changes not captured by smooth scaling alone; GPT-3 introduced in-context learning through prompt conditioning without parameter updates.The text leaves open whether such effects are phase transitions, evaluation artifacts, or a mixture.Source: §3.16.2 · pp. 73–74
  • Possible explanations for scale-dependent behavior include changing operator rank, richer feature directions, more stable circuits, and stronger geometric effects, but these remain hypotheses.Scaling laws do not yet provide a mechanistic account of their regularities or explain why some capabilities emerge abruptly.Source: §3.16.2 · p. 73
  • Scaling behavior has also been observed in contrastive language-image pretraining and multimodal generative models, extending the reported regularity beyond text-only models.The section frames this as a broader empirical phenomenon across large neural operator systems.Source: §3.16.2 · p. 73

4 Exploiting Spectral Characteristics of Transformers

This section surveys numerical linear algebra techniques for exploiting spectral structure to reduce the cost of LLM fine-tuning and transformer attention, including compression and low-rank methods. It also describes FAVOR, a random-feature approximation to attention, its computation, and its complexity and rank properties.

4.1 Model Compression

  • Compression reduces storage and computation while aiming to preserve accuracy through pruning, quantization, knowledge distillation, and low-rank factorization.Pruning removes unnecessary connections and retrains remaining weights; quantization enables integer-only inference, while distillation trains a student to match a teacher’s softened outputs [64].Source: §4.1 · p. 74, §4.1.1 · p. 74, §4.1.2 · p. 75

4.2 LoRa: Exploiting Low-rank structure in LLMs

  • LoRa [67] keeps pretrained weights fixed and trains low-rank factors, with the LLM adaptation suggested for self-attention weights Wq, Wk, and Wv.The motivation is that near convergence, Hessian eigenvalues cluster near zero and parameter-space dimension can be small even in overparameterized networks.Source: §4.2 · p. 77
  • LoRa reduced the parameters needed to fine-tune GPT3 from 175B to 17M, a reported 10,000-fold gain.The example is presented as an illustrative result.Source: §4.2 · p. 77

4.3 The idea of ‘linear’ transformers

  • Standard attention forms an n × n score matrix at O(n^2) cost, while linearized attention reorders products to avoid forming that matrix.With softmax omitted, the reordered computation is O(n); softmax-decoupling alternatives approximate the original method, which is described as fairly accurate in practice.Source: §4.3 · pp. 78–79

4.4 Randomization and random projections approaches

  • Linformer [146] uses low-rank attention approximations and random projections, while Performer [30] uses random features to approximate attention kernels.These methods exploit the observation that real-world data often has low effective rank because of correlation or low-dimensional structure.Source: §4.4 · pp. 80–81
  • The practical approximation uses one sampled query-key feature pair corresponding to M random feature functions, rather than computing the exact expectation.Source: §4.4 · p. 82
  • To compute the attention output, the method forms C = [V, 1_n], applies sampled query and key features, multiplies through X and Y, then normalizes Y to obtain the approximation to AV.The normalization uses D = diag(Y(:, d + 1)) and returns D^-1Y(:, 1:d).Source: §4.4 · p. 82
  • FAVOR has O(nMd) time complexity, O(Md + nd + Mn) memory complexity, and produces a rank-M approximation to the attention matrix.The described algorithm covers bidirectional attention without masking; the authors also present a unidirectional version.Source: §4.4 · p. 82

5 Advanced optimization for Transformers

The section examines the difficulty of exact second-order optimization in deep learning and surveys practical curvature-based alternatives for Transformers, including natural gradients, Gauss–Newton connections, K-FAC, Shampoo, and Muon. It also discusses Shampoo’s gradient-scaling approach and reported convergence, runtime, and competition results, alongside Muon’s Newton–Schulz preconditioning.

5 Advanced optimization for Transformers

  • Nonconvexity, stochastic gradients, and Hessian computation and storage make second-order methods costly, potentially unstable, or ineffective for deep learning.Erratic minibatch samples also make cost-effective Hessian approximation difficult.Source: §5 · p. 83

5.1 Natural Gradients

  • Natural gradient descent replaces Euclidean geometry with a parameter-dependent Riemannian metric, yielding a Fisher-inverse gradient direction for predictive-distribution KL.The Fisher matrix captures the local quadratic approximation of the predictive-distribution divergence.Source: §5.1 · pp. 83–86
  • Natural gradient descent converged in fewer than 100 iterations, while standard gradient descent failed to reach the optimum in 10,000 steps.For the direction-distorted Gaussian example, the Fisher-metric direction points toward the solution; with an optimal step size of 1, it could converge in one step.Source: §5.1 · pp. 86–87, Figure 30 · p. 87

5.2 Connection with the Gauss-Newton method

  • The generalized Gauss–Newton matrix extends least-squares curvature to empirical model losses and equals the Fisher for a broad class of exponential-family objectives.In least squares, the Gauss–Newton approximation is justified near a solution when residuals are small.Source: §5.2 · p. 88

5.3 The K-FAC approach

  • K-FAC exploits layer structure to approximate the Fisher with Kronecker-factored blocks, whose inverse separates into inverses of smaller factors.The full Fisher is too costly to compute; the factorization is an approximation, not an equality in general.Source: §5.3 · pp. 88–89

5.4 The Shampoo approach

  • Shampoo avoids K-FAC’s layer-specific structure by preconditioning matrix gradients with left and right second-moment factors, using m^2 + n^2 storage.Its updates resemble and generalize AdaGrad’s diagonal preconditioner.Source: §5.4 · pp. 89–90
  • Shampoo accumulates dense left- and right-sided gradient matrices, then scales the gradient on each side by the corresponding accumulated matrix raised to −1/4.Source: §5.4 · p. 90
  • Shampoo is reported to converge considerably faster than commonly used optimizers and to achieve much shorter runtime than vanilla SGD.The passage attributes the lower cost to fractional-power operations on relatively small matrices, despite the more complex update rule.Source: §5.4 · p. 90
  • A distributed Shampoo algorithm won the 2024 AlgoPerf Training Algorithms Competition, demonstrating superiority over AdamW.Source: §5.4 · p. 91

5.5 Muon

  • Muon was derived to avoid Shampoo’s main drawback of using two scaling matrices, approximating semi-orthogonal preconditioning with Newton–Schulz iteration on momentum.For a square gradient, the target is the semi-orthogonal matrix UV^T from its SVD; Muon instead iteratively approximates the corresponding factors for momentum Mt.Source: §5.5 · p. 91

6 Concluding remarks

AI research is advancing so quickly that survey coverage can become obsolete, while impactful work often depends on statistical intuition, extensive testing, and large teams and machines. The authors urge NLA specialists to engage with AI, where many innovations use NLA concepts.

6 Concluding remarks

  • 9,122, 10,411, 12,345, 15,671, and 21,575 NeurIPS submissions from 2021 to 2025 illustrate rapid growth, making it difficult to keep abreast of developments and leaving survey topics vulnerable to obsolescence.The 2025 count was 40.6% above 2024, which itself increased by approximately 27% over the previous year.Source: §6 · pp. 91–92
  • The 2024 MLCommons announcement said non-diagonal preconditioning had “dethroned Nesterov Adam.”Source: §6 · p. 91
  • AI research often relies on statistical intuition followed by extensive testing on big parallel machines, putting traditional NLA teams at a disadvantage.The authors say big teams and big machines are mandatory for truly impactful innovations such as Transformers.Source: §6 · p. 92
  • The authors argue that NLA specialists should participate in AI research because many innovative efficient variants mainly use NLA concepts.They note that learning is needed, but say abundant resources and AI itself can facilitate it.Source: §6 · p. 92
Loading 2610.04631v1…