PaperlayerView exact revision on arXiv

Source-linked AI summary

Transformers as Support Vector Machines

Davoud Ataee Tarzanagh, Yingcong Li, Christos Thrampoulidis, Samet Oymak

arXiv:2308.16898v3cs.LGcs.AIcs.CLmath.OC

TL;DR

The paper studies how attention selects and composes tokens and how its optimization geometry shapes implicit bias. It formulates attention as a hard-margin SVM, analyzes convergence across parameterizations, and reports max-margin convergence alongside conditions that support global convergence.

  • Problem

    The paper asks how to characterize transformers' optimization landscape and implicit bias, including how attention selects and composes tokens.

    Source: §1 · p. 1

  • Method

    It connects attention optimization to hard-margin SVMs that separate optimal from non-optimal tokens, then extends the equivalence to nonlinear prediction heads.

    Source: §1 · p. 3

  • Results

    Without early stopping, gradient descent converges toward maximum-margin solutions for the stated decreasing losses; over-parameterization can also catalyze global convergence.

    Source: §1 · p. 3, §9 · p. 19

  • Takeaways & Limitations

    The findings support viewing self-attention as a mechanism that separates and selects optimal tokens through hard-margin geometry.

    Source: §9 · p. 19

    The theorem requires a unique optimal token per input, a condition the paper says holds for almost all datasets after slight feature perturbations.

    Source: §3 · p. 7

Abstract

from arXiv · show

Since its inception in "Attention Is All You Need", transformer architecture has led to revolutionary advancements in NLP. The attention layer within the transformer admits a sequence of input tokens $X$ and makes them interact through pairwise similarities computed as softmax$(XQK^\top X^\top)$, where $(K,Q)$ are the trainable key-query parameters. In this work, we establish a formal equivalence between the optimization geometry of self-attention and a hard-margin SVM problem that separates optimal input tokens from non-optimal tokens using linear constraints on the outer-products of token pairs. This formalism allows us to characterize the implicit bias of 1-layer transformers optimized with gradient descent: (1) Optimizing the attention layer with vanishing regularization, parameterized by $(K,Q)$, converges in direction to an SVM solution minimizing the nuclear norm of the combined parameter $W=KQ^\top$. Instead, directly parameterizing by $W$ minimizes a Frobenius norm objective. We characterize this convergence, highlighting that it can occur toward locally-optimal directions rather than global ones. (2) Complementing this, we prove the local/global directional convergence of gradient descent under suitable geometric conditions. Importantly, we show that over-parameterization catalyzes global convergence by ensuring the feasibility of the SVM problem and by guaranteeing a benign optimization landscape devoid of stationary points. (3) While our theory applies primarily to linear prediction heads, we propose a more general SVM equivalence that predicts the implicit bias with nonlinear heads. Our findings are applicable to arbitrary datasets and their validity is verified via experiments. We also introduce several open problems and research directions. We believe these findings inspire the interpretation of transformers as a hierarchy of SVMs that separates and selects optimal tokens.

1 Introduction

The paper asks how attention selects and composes tokens and what optimization geometry and implicit bias govern that process. It connects attention to hard-margin SVMs, distinguishing parameterizations and analyzing when gradient descent converges globally or locally.

1 Introduction

  • The framework connects attention optimization to a hard-margin SVM that separates and selects optimal tokens using linear constraints.The paper reports that its findings apply to arbitrary datasets whenever the SVM is feasible and are numerically verifiable.Source: §1 · pp. 1, 3
  • Vanishing-regularization optimization of (K, Q) converges directionally to a nuclear-norm SVM solution, whereas direct W parameterization yields a Frobenius-norm solution.The distinction reveals a low-rank bias for the factorized parameterization.Source: §1 · p. 2
  • For combined key-query variable W, gradient descent can converge to locally optimal SVM directions, while geometric conditions can guarantee the globally optimal direction.Local optimality requires selected tokens to score higher than their neighboring tokens.Source: §1 · p. 2
  • The analysis fixes the prediction head and optimizes only attention weights, avoiding zero training loss achieved through the head alone.The authors give neural tangent kernel arguments as an example of the degenerate case they seek to avoid.Source: §1 · p. 2
  • Over-parameterization can ensure SVM feasibility and a benign landscape without stationary points or spurious locally optimal directions, catalyzing global convergence.The paper identifies these as geometric conditions for global convergence.Source: §1 · p. 3
  • For nonlinear heads, the proposed generalized SVM equivalence predicts gradient-descent bias by separating token selection from the composition of selected tokens.Its directional component applies a 0-1 selection mask, while a finite component adjusts softmax probabilities.Source: §1 · p. 3

2 Preliminaries

The preliminaries set up regularized paths and gradient descent for attention weights, then define optimal tokens and the associated hard-margin SVM problems. These formulations distinguish the geometry of direct W optimization from factorized (K, Q) optimization.

2 Preliminaries

  • The regularized path tracks solutions as a norm bound grows, and its directional limit is used to study gradient descent's implicit bias.The paper also notes that local regularized paths can examine bias along local directions.Source: §2 · p. 4

2.1 Optimal tokens and hard-margin SVM problem for cross attention

  • An optimal token maximizes the label-weighted score γ_it = Y_i · v^⊤x_it within its input sequence.The setup uses a linear prediction head for most of the theoretical exposition.Source: §2 · p. 3, §2.1 · p. 4
  • (Att-SVM) separates each optimal token from the other tokens through constraints on their query-weighted similarities.Its constraints express that the optimal token's key-query feature score exceeds each alternative by at least one.Source: §2.1 · pp. 4–5
  • For almost all datasets, (Att-SVM) is feasible when d ≥ max(T − 1, n), including the self-attention setting.The condition is stated for the token-separation problem across the inputs.Source: §2.1 · p. 5
  • The convex (Att-SVM) does not fully capture gradient-descent geometry, which can converge to an SVM solution over locally optimal tokens.When the attention output must choose one optimal token, softmax probabilities approach a one-hot vector as the weight norm diverges.Source: §2.1 · p. 5
  • Factorizing W as KQ adds nonconvexity; with full parameterization m ≥ d, its rank constraint disappears and the nuclear-norm problem is convex.The nuclear norm encourages low-rank solutions, and an optimal solution to either stated W-SVM has rank at most n.Source: §2.1 · p. 5

3 Understanding the Implicit Bias of Self-Attention

Regularization-path analysis characterizes attention’s implicit bias: factorized key-query parameters and direct W parameterization converge toward SVM solutions with different norm objectives. The section also relates optimal-token selection to training risk and distinguishes global regularization-path geometry from gradient descent’s local behavior.

3 Understanding the Implicit Bias of Self-Attention

  • When optimal tokens are not all tokens, training risk exceeds the optimal-token loss, while a feasible separator scaled without bound approaches that loss.The result assumes a strictly decreasing loss with bounded derivative; examples include logistic, exponential, and correlation losses.Source: §3 · p. 6
  • Factorized (K, Q) regularization paths converge toward a max-margin Att-SVM⋆ solution minimizing the nuclear norm of W = KQ⊤, while direct W paths use a Frobenius-norm Att-SVM objective.The factorized parameterization therefore has a low-rank bias; with m = d, Att-SVM⋆ is convex without rank constraints.Source: §3 · p. 6
  • Regularization paths capture global geometry, but do not characterize gradient descent’s bias toward locally optimal directions; the stated theorem requires a unique optimal token per sequence.The text says this uniqueness condition holds for almost all datasets when input features are slightly perturbed.Source: §3 · p. 7
  • In the illustrated two-input cross-attention setting, gradient trajectories align incrementally with their respective Att-SVM and Att-SVM⋆ directions.The figure uses n = d = m = 2 and T = 5; the text presents the alignment as support for Theorem 2.Source: §3 · p. 7
  • Att-SVM⋆ imposes rank(W) ≤ m, but the rank constraint is inconsequential when the unconstrained problem has a low-rank solution.The paper cites a rank-at-most-n bound for optimal Att-SVM and Att-SVM⋆ solutions.Source: §3 · p. 7

4 Global Convergence of Gradient Descent

The section identifies favorable initial gradients and sufficient overparameterization as routes to global convergence, while establishing conditions on the optimization landscape.

4.2 Provable global convergence of 1-layer transformer

  • The section identifies sufficiently large d as a catalyst for global convergence, and reports that overparameterization increasingly supports the globally optimal max-margin direction.The authors connect dimension growth to Assumption B.1 and to the norm-divergence prerequisite for convergence.Source: §4 · p. 7, §4.2 · p. 10

4.1 Properties of optimization landscape

  • Under Assumptions A and B, GD parameters diverge in norm; SVM feasibility is necessary for convergence to the Wmm direction.The theorem’s landscape criteria include absence of stationary points and divergence of the parameter norm.Source: §4.1 · p. 8

4.2 Provable global convergence of 1-layer transformer

  • A separating first gradient step yields a cone without stationary points; if all iterates remain within it, they diverge in norm and converge directionally to Wmm.The directional-convergence result requires the iterates to remain in the specified cone.Source: §4.2 · p. 9
  • As d increases, global-convergence rates approach 100% in Figure 5 and reach 100% in Figure 6 for both parameterizations.Figure 5 also reports a higher probability of Assumption B.1 as d grows.Source: Figure 5 · p. 8, Figure 6 · p. 9, §4.2 · p. 10

5 Understanding Local Convergence of 1-Layer Transformer

Gradient descent can converge toward locally optimal token selections, not only the globally optimal direction. The section gives geometric conditions for local and global convergence and studies how dimension and SVM margins relate to those outcomes.

5.1 Local convergence of gradient descent

  • Locally optimal token selections can attract gradient descent: from suitable initializations, iterates diverge in norm and converge directionally to the corresponding SVM solution.The theorem also gives a cone around that solution containing no stationary points.Source: §5.1 · p. 11

5.2 Overparameterization conjecture: When local-optimal directions disappear

  • Under Assumptions B.1 and D, the global SVM direction is the only viable local direction; for almost all datasets satisfying these conditions, GD converges globally with a proper constant learning rate.Assumption D rules out locally optimal alternatives, while the section presents its combined conditions as a global-convergence conjecture.Source: §5.2 · pp. 11–12

5.3 Investigation on SVM objectives and GD convergence

  • When the global SVM margin is small or infeasible, GD tends to converge to a local direction with a comparatively larger margin.Figure 8 contrasts margins for globally optimal and locally optimal token indices with global and local convergence frequencies.Source: Figure 8 · p. 12, §5.2 · p. 12, §5.3 · p. 13

5.4 Guarantees on local regularization path

  • When the norm objective is not strongly convex, the SVM solution set may not be a singleton.The appendix generalizes regularization-path results to sequence-to-sequence classification and a class of monotonicity-preserving prediction heads.Source: §5.4 · pp. 13–14
  • When d ≥ max(T − 1, n) and m = d, the SVM is feasible for almost all datasets and every choice of token indices, including in self-attention.At m = d the rank constraint is vacuous; Frobenius- and nuclear-norm specializations recover the corresponding earlier regularization-path results.Source: §5.4 · pp. 13–14
  • The local regularization path constrains optimization to a score-based cone where lower-score tokens can block movement toward higher-score tokens.For locally optimal indices, the path converges toward the local SVM direction; for unique globally optimal indices, the cone can be the full rank-≤m matrix set.Source: §5.4 · pp. 13–14

6 Toward A More General SVM Equivalence for Nonlinear Prediction Heads

For nonlinear prediction heads, attention may compose multiple tokens, motivating an SVM formulation that predicts both their composition and selection. Experiments report that this generalized equivalence aligns with gradient-descent solutions, while increased multi-token selection can reduce predictivity.

6 Toward A More General SVM Equivalence for Nonlinear Prediction Heads

  • The generalized SVM separates selected from masked tokens with a margin while requiring equal similarity among selected tokens, and combines this directional component with a finite component that sets their softmax probabilities.The formulation minimizes the Frobenius norm for the directional constraints.Source: §6 · pp. 14–15
  • The generalized equivalence aligns better with GD than the one-token baseline, and its average correlation exceeds 0.99 in the larger-d regime.The equivalence including the finite-component correction further improves correlation relative to the Gen-SVM direction.Source: Figure 9 · p. 15, §6 · p. 16

6.1 When does attention select multiple tokens?

  • For orthonormal tokens under the stated score-gap condition, the optimal softmax selection contains at least two and at most τ nonzero entries.This establishes a controllable multi-token selection example for a concave prediction score.Source: §6.1 · p. 16
  • As λ increases, more tokens are selected; λ = 0 selects one token, while λ > 2.0 selects all tokens under the experiment’s criteria.The SVM equivalence remains consistently above 0.95 correlation with GD, but predictivity decreases as more tokens are selected.Source: §6.1 · p. 17, Figure 10 · p. 17
  • The nonlinear-head experiments are an initial approach, with filtering and imperfect correlations identified as caveats requiring further investigation.Source: §6.1 · p. 17

7 Extending the Theory to Sequential and Causal Predictions

The paper extends its regularization-path guarantees to sequence-to-sequence prediction and adapts the SVM constraints to causal and general attention masks. The causal constraints restrict each prediction to comparisons among tokens available at that position.

7 Extending the Theory to Sequential and Causal Predictions

  • The sequence-to-sequence extension retains the regularization-path guarantees after generalizing Att-SVM to train and classify all T tokens per input.The authors state that the guarantees are rigorously established in Appendix E.Source: §7 · p. 17
  • The sequential SVM formulation imposes margin constraints for every token position k and every competing token t in the input.The displayed constraints compare selected token indices α_ik against t for each k.Source: §7 · p. 18
  • For causal attention, SVM constraints apply only to tokens at positions t≤k, so prediction k selects from the first k tokens.This follows the causal mask, which prevents a prediction from using future tokens.Source: §7 · pp. 17–18
  • General attention masks can be handled by enforcing SVM constraints only on their nonzero support, and Gen-SVM extends the framework to composing multiple tokens.Causal attention is a special case of such a mask.Source: §7 · p. 18

8 Related work

The related work situates the paper's attention-specific max-margin analysis among research on implicit bias, sparsity, and transformer mechanisms. It distinguishes the proposed token-separation framework from prior analyses of classification, matrix factorization, and transformer optimization.

8.1 Implicit regularization, matrix factorization, and sparsity

  • Earlier research studies implicit bias in separable and non-separable classification, regression, stochastic optimization, and adaptive or momentum-based methods.The cited works include [SHN+18] [GLSS18] [NLG+19] [MWG+20] [JT20] on classification and [GLSS18] [YKM20] [VKR19] [AW20b] [SATA22] on regression.Source: §8.1 · p. 18
  • Unlike prior max-margin analyses of classification and matrix factorization, this work analyzes a nonconvex attention problem that separates tokens and adds guarantees for global convergence, over-parameterization, and nonlinear heads.The authors identify [TLZO23] as the closest prior attention–SVM work and describe their transformer guarantees as extensions beyond its simpler attention model.Source: §8.1 · p. 18
  • Softmax and lasso share feature-selection parallels, while Att-SVM selects one token and Gen-SVM models broader token-selection sparsity.Softmax outputs have ℓ1 norm 1 and can approach one-hot vectors as weights scale up, unless all logits are equal.Source: §8.1 · pp. 18–19

8.2 Attention mechanism and transformers

  • For linear heads, attention dynamics collapse onto one token; the paper contrasts this with nonlinear heads, which provably require selecting and composing multiple tokens.The discussion connects this contrast to the expressivity of transformer MLP layers.Source: §8.2 · p. 19
  • The paper relates its account of sparse attention and low-rank updates to prior empirical and theoretical studies of attention weights [MRG+20] [BALA+23].Its KQ-SVM formulation incorporates low-rank structure through a nuclear-norm objective, a rank-m constraint, and an implicit constraint.Source: §8.2 · p. 19
  • Prior transformer theory often relies on strict statistical assumptions, whereas this optimization-theoretic analysis links attention to max-margin SVM geometry across diverse datasets.The authors present the equivalence as a framework for further research.Source: §8.2 · p. 19

9 Discussion, Future Directions, and Open Problems

The discussion frames self-attention through hard-margin SVM geometry and summarizes gradient descent's maximum-margin behavior. It identifies open questions on convergence, generalization, joint optimization, and extending the theory to realistic architectures.

9 Discussion, Future Directions, and Open Problems

  • Self-attention's optimization geometry connects to hard-margin SVMs, whose linear constraints on token-pair outer products separate optimal from non-optimal tokens.The paper presents this connection as a central finding of its framework.Source: §9 · p. 19
  • Without early stopping, gradient descent converges toward a maximum-margin solution under logistic, exponential, or other smooth decreasing losses.The stated implicit bias is independent of initialization and unaffected by sufficiently small step sizes that permit convergence.Source: §9 · p. 19
  • Non-asymptotic convergence rates remain open; analyzing them could inform choices of learning rate, initialization, and optimization method.The paper establishes asymptotic convergence of gradient descent.Source: §9 · p. 20
  • The paper leaves generalization guarantees for gradient-based algorithms as future work, noting that the SVM connection may facilitate their analysis.The authors compare this potential role to kernel/NTK analysis for deep nets.Source: §9 · p. 20
  • A complete account of gradient descent's directional convergence from arbitrary initialization in one-layer self-attention remains open.The discussion points to over-parameterization and locally optimal directions as relevant pieces of the problem.Source: §9 · p. 20
  • Extending the max-margin equivalence to multi-head, multi-layer, and MLP-nonlinear architectures remains a research direction.The authors describe the nonlinear-head results in Section 6 as a step toward that goal.Source: §9 · p. 20
  • Jointly optimizing attention weights and the prediction head remains an open problem, framed as low-rank factorization with W passing through softmax.The paper notes preliminary regularization-path geometry for a simpler attention model in [TLZO23].Source: §9 · p. 20

A Proof of Theorem 1: Separability Under Mild Over-Parameterization

The proof establishes SVM separability under mild over-parameterization by showing that the relevant token-pair features are linearly independent almost surely. It also uses projection arguments to preserve constraints while controlling the nuclear norm.

A Proof of Theorem 1: Separability Under Mild Over-Parameterization

  • The SVM constraints require ⟨w,f_it⟩≥1 for each example and competing token, with f_it=(x_iα_i−x_it)⊗z_i.The proof states that these constraints are feasible when the feature vectors are linearly independent.Source: A Proof of Theorem 1: Separability… · p. 27
  • For a fixed choice of selected indices, the proof reduces feasibility to feature independence and uses a union bound over the finite set of choices.This establishes the result for all such choices once almost-sure separation is shown for an arbitrary fixed choice.Source: A Proof of Theorem 1: Separability… · p. 27
  • When d≥max(T−1,n), the perturbed feature matrix for each example has rank T−1 almost surely, supporting the induction to full row rank.The proof inductively establishes full row rank after random Gaussian perturbation.Source: A Proof of Theorem 1: Separability… · p. 28

B.1 Proof of Lemma 1

  • In the Lemma 1 proof, projecting W_mm onto the specified subspace preserves the SVM constraints and does not increase its nuclear norm.The proof compares the projected matrix's nuclear norm with that of W_mm.Source: §B.1 · p. 29

B.2 Proof of Lemma 2

The proofs establish how feasible token separation shapes the loss and derive gradient conditions for norm growth and directional convergence, including global convergence under suitable geometry.

B.2 Proof of Lemma 2

  • When the attention SVM is feasible, scaling a separating matrix makes the softmax select an optimal token and the loss approach its optimum.If any token is non-optimal, finite softmax weights give strictly greater loss than the optimal-token loss.Source: §B.2 · p. 29

B.3 Proof of Lemma 4

  • The proof derives gradient expressions and bounds for the combined parameter and its key-query factors under the loss assumptions.The bounds use an inequality for products and Assumption A.Source: §B.3 · pp. 30–31

B.4 A useful lemma for gradient descent analysis

  • Lemma 8 introduces a reusable setup involving attention weights, token scores, and an alternate value matrix.It defines a = XVz, s = S(XWz), and γ = Xv.Source: §B.4 · p. 31

C.1 Divergence of norm of the iterates

  • With a sufficiently small step size, gradient descent decreases the loss, while its gradient has strictly negative correlation with the Att-SVM solution.The negative correlation holds for every finite W under Assumptions A and B, implying the iterates' norms diverge.Source: §C.1 · p. 32, §C.1.1 · p. 33

C.2 Global convergence under good initial gradient

  • Under the stated geometric conditions, the cone contains no stationary points, supporting convergence of gradient directions to the max-margin solution.The no-stationary-point claim holds for a sufficiently large radius.Source: §C.2 · pp. 34–35
  • As π approaches zero, within the specified cone the negative gradient becomes more correlated with the max-margin solution than with W itself.The result applies for sufficiently large norm within the cone.Source: §C.2 · p. 35

C.2.1 Proof of Theorem 4

Theorem 4 establishes directional convergence to the Att-SVM solution when gradient descent starts in and remains within a specified cone; it also gives conditions ensuring that iterates stay there.

C.2.1 Proof of Theorem 4

  • For any µ > 0, a sufficiently large radius makes the associated cone free of stationary points.This is the theorem's L1 claim.Source: §C.2.1 · p. 38
  • If the iterates remain in the specified cone, their norms diverge and their normalized directions converge to the normalized Att-SVM solution.The theorem assumes zero initialization followed by a sufficiently large initial step and subsequent stepsizes η ≤ 1/LW.Source: §4.2 · p. 9, §C.2.1 · pp. 38, 41
  • Under the theorem's step-size and cone conditions, all gradient-descent iterates remain within the cone.The proof establishes this invariance after entering the specified region.Source: §C.2.1 · pp. 38–39

D Local Convergence of Gradient Descent

This section analyzes local gradient-descent convergence around a max-margin SVM solution, including conditions for directional convergence and a sequence-to-sequence extension of empirical risk minimization. It also establishes a one-to-one correspondence between the regularization paths of the direct and factorized parameterizations, with equal optimal costs.

D Local Convergence of Gradient Descent

  • For sufficiently large radius, Lemma 13 establishes a cone around the Frobenius-norm Att-SVM solution that contains no stationary points.The lemma assumes the stated loss condition and locally optimal tokens.Source: D Local Convergence of Gradient Descent · p. 41

D.1 Proof of Theorem 5

  • With the prescribed initialization and a suitable constant step size, gradient descent stays in the cone, its norm diverges, and its normalized iterates converge to the normalized max-margin SVM solution.The proof shows that iterates eventually enter arbitrarily narrow cones around the solution direction.Source: §D.1 · pp. 47, 49

E Convergence of Regularization Path for Sequence-to-Sequence Setting

  • SERM generalizes empirical risk minimization to two input sequences, cross-attention producing K tokens, and K separate prediction heads.The formulation allows regression of multiple token outputs.Source: E Convergence of Regularization Path… · p. 49
  • For every R >= 0, regularization-path solutions in W and factorized (K,Q) parameterizations map one-to-one and have equal optimal costs.For any solution in either parameterization, a corresponding solution exists in the other.Source: E Convergence of Regularization Path… · p. 50

E.1 Local regularization path and proof of Theorem 7

The local regularization path converges toward locally optimal directions under the stated feasibility and prediction-head conditions. When the selected tokens are uniquely optimal, the cone becomes the full parameter space, connecting local-path convergence to global convergence.

E.1 Local regularization path and proof of Theorem 7

  • The cone prioritizes lower-scoring tokens, which create a barrier that can prevent optimization from moving toward higher-scoring tokens.The local path minimizes loss over a norm-constrained cone associated with the selected tokens.Source: §E.1 · p. 51
  • The cone-restricted regularization path converges directionally to locally optimal SVM directions when the SVM is feasible and Assumptions A and E hold.The theorem considers sufficiently small ε and sufficiently large radius R.Source: §E.1 · pp. 51–52

E.2 Global regularization path

  • When optimal token indices are unique, the cone is the full parameter space, so the convergence guarantee applies to globally optimal directions.The global-path corollary additionally assumes Assumptions A and E.Source: §E.1 · p. 51, §E.2 · p. 55

E.1 Local regularization path and proof of Theorem 7

  • Assumption E requires the prediction head to preserve token-score ordering under convex combinations, and applies over the training dataset.The condition strictly generalizes linearity.Source: §E.1 · p. 51

E.2 Global regularization path

  • The corollary extends the result to (SERM-W) and (SERM-KQ), recovering Theorem 2 for K = 1 and m = d.Source: §E.2 · p. 55

E.2.1 Proof of Theorem 2

The proof analyzes regularization paths by comparing their loss and margin behavior with SVM solutions. It establishes directional convergence claims and relates the KQ parameterization’s alignment with the SVM solution to its rank limit.

E.2.1 Proof of Theorem 2

  • The SVM solution separates optimal tokens from the remaining tokens, yielding softmax probability bounds used to compare path losses.The bounds in the proof contrast probabilities under the SVM reference solution and a path parameter that violates a margin constraint.Source: §E.2.1 · pp. 55–56
  • The proof shows that the regularization path’s norm diverges and that its direction converges to the set of SVM minima.The argument uses a contradiction based on a candidate direction that violates an SVM margin constraint.Source: §E.2.1 · p. 56
  • The reported subfigures, with n fixed at 5 and T fixed at 5, show that as m approaches or exceeds n, KQ⊤ aligns more with Wmm.Source: §E.2.1 · p. 57

F Supporting Experiments

The supporting experiments test nonlinear-head alignment, rank sensitivity, and multi-token selection. They report strong alignment under specified dimension or masking conditions, and selected-token counts bounded by τ.

F Supporting Experiments

  • For Figures 2, 5, and 6, experiments enforce a minimum score gap of γ = 0.1 because PyTorch may not select optimal tokens when gaps are too small.Source: F Supporting Experiments · p. 58
  • When m ≳ n, W converges directionally toward Wmm⋆,α; performance remains consistent across the tested T values.The experiments vary m from 1 to 10, with n or T varied in separate subfigures.Source: F Supporting Experiments · p. 58
  • For d ≥ 8 or Γ ≤ 10^-9, GD’s W achieves correlation > 0.99 with the SVM-equivalence WSVMeq.Figure 12 reports the dimension and masked-token comparisons.Source: F Supporting Experiments · p. 58
  • Figure 13’s selected-token counts remain within τ, and the reported counts correlate positively with τ and λ.The figure’s distribution covers τ ∈ {3, 5, 7, 9}.Source: F Supporting Experiments · p. 58, Figure 13 · p. 58
Loading 2308.16898v3…