PaperlayerView exact revision on arXiv

Source-linked AI summary

Context Staircase: Signature-Aligned Dynamics of Token Embeddings under Small Initialization

Junjie Yao, Liangkai Hang, Zhi-Qin John Xu

arXiv:2608.30315v1cs.LGcs.CL

TL;DR

How meaningful token-embedding structure emerges during training, and whether its changing geometry reflects identifiable data statistics and model architecture, remains unclear. This paper formalizes token-conditioned probability signatures and studies embedding evolution under small initialization, finding early alignment with low-order signatures followed by jointly emerging higher-order structures.

  • Problem

    It remains unclear how embedding structures emerge during training or whether their changing geometry relates to identifiable data statistics and model architecture.

    Source: §1 · p. 2

  • Method

    The paper defines token-conditioned probability signatures and investigates embedding evolution under small initialization across controlled sequence-prediction tasks.

    Source: §1 · p. 2

  • Results

    Embeddings align early with low-order signatures, while multiple higher-order contextual signatures later become jointly visible, with the highest accessible order strongest under analyzed late-stage conditions.

    Source: §1 · p. 3, §8 · p. 37

  • Takeaways & Limitations

    The signature perspective connects embedding geometry with interpretable gradient-flow dynamics and describes how architectural paths modulate structures and when they become useful for task solving.

    Source: §8 · p. 38

    Signature-aligned geometry alone does not ensure task success; usefulness depends on whether computations after embedding lookup can exploit the encoded structure.

    Source: §8 · pp. 37–38

Abstract

from arXiv · show

Token embeddings are the basic representational units that connect discrete tokens with continuous computation in language models. Although modern language models learn embeddings from random initialization through gradient-based training, the dynamical mechanism by which meaningful embedding structures emerge remains unclear. In this work, we identify that the evolving embedding structures are closely related to token-conditioned label and contextual distributions, which we formalize as probability signatures. We observe a progressive learning process, which we term Context Staircase: embeddings learn the low-order statistic signatures of the data before the high-order ones. More specifically, we observe that early in training they align with the simplest, context-free signature linking a token to its label, and as training proceeds, they progressively reflect signatures involving more and more context tokens. We then analyze the gradient flow of embeddings under small initialization to explain this phenomenon, deriving embedding evolution equations for feed-forward and self-attention architectures. We further extend these observations to real language-model training. Finally, we show that these embedding structures play an important role in both task learning and the incorporation of semantic structure into the embedding space. Overall, our results provide a dynamic explanation of how data statistics and architecture jointly shape token embeddings in language models, and reveal an implicit bias in the space of data statistics: training proceeds from simpler, low-order statistical relations toward increasingly complex, context-dependent ones.

1 Introduction

The paper asks how training writes data statistics into token embeddings and argues that, under small initialization, simpler signatures emerge before increasingly context-dependent ones. It links this progression to gradient dynamics, model architecture, and the role of embedding structure in task learning and semantic organization.

1 Introduction

  • Under small initialization, embeddings first align with low-order probability signatures, then increasingly reflect higher-order, context-dependent signatures as training proceeds.In the addition task, embeddings first form a linear structure matching the low-order signature, then develop an odd–even separation reflected in the higher-order signature.Source: §1 · pp. 2–4
  • Small initialization suppresses higher-order gradient contributions early; as parameter scale grows, accessible orders reach the same leading scale, with the highest-order contribution largest under late-stage conditions.The gradient decomposes into contributions associated with label signatures of different orders, providing a dynamical explanation for the progression from simpler to more context-dependent structures.Source: §1 · pp. 2–4
  • In self-attention, signature terms depend on position and are modulated by attention scores; the analysis traces computational paths in decoder-only Transformers and tests hypotheses on real language-model training.Source: §1 · pp. 3–4
  • Signature-aligned geometry can help a model learn a numerical comparison rule, while corpus-derived signatures show semantic organization in numbers, weekdays, and noun categories.The comparison task demonstrates learning once the embedding space acquires the structure encoded by its corresponding signature.Source: §1 · pp. 3–4

2 Related Work

Prior work links embedding geometry to text statistics and studies initialization-dependent dynamics, but leaves unclear which data statistics token embeddings progressively acquire; this work addresses that gap through probability signatures.

Token embeddings and representation geometry

  • Classical embedding methods tied vector geometry to contextual distributions, while later contextual-model work documented anisotropy, recoverable syntax, and structured semantic or world-knowledge representations.This literature spans trainable language-model embeddings, local context prediction, global co-occurrence objectives, and subword representations.Source: §2 · p. 4
  • Studies find that initialization scale shapes representation structures and learning behavior, with small initialization associated with more structured spaces, reasoning, and compositional generalization.The cited work identifies initialization as a factor in how embedding structure emerges during training.Source: §2 · p. 4

Training dynamics and gradient-flow analyses

  • This work analyzes token embeddings under small initialization, identifying a hierarchy of data-derived probability signatures in their evolving geometry.The analysis connects the embedding updates to signatures entering through different computational paths.Source: §2 · p. 5
  • Gradient-flow and Transformer studies examine small-initialization feature learning, attention dynamics, and corpus-dependent parameter formation, but chiefly analyze network weights rather than token embeddings.Related analyses include condensation and staged attention development; Im et al. (2026) derive Transformer-weight characterizations using corpus-dependent basis functions such as bigram and context mappings.Source: §2 · p. 5

3 Preliminary

The preliminary section defines token prediction, a hierarchy of token-conditioned probability signatures, and the small-initialization regime used to explain Context Staircase: embeddings move from low-order statistical structure toward richer contextual structure during training.

3.1 Prediction Problem

  • The setup models a sequence X and target token y over a finite vocabulary, using input embeddings and an output unembedding to predict y with softmax cross-entropy.The paper studies input-embedding gradient flow to relate data distributions and model components to embedding updates and geometry.Source: §3.1 · p. 6

3.2 Token-conditioned Signatures

  • For each token, the signature hierarchy includes conditional label and context co-occurrence/label distributions; marginalizing a context mode recovers the corresponding lower-order signature.The initial analysis assumes exchangeable context positions, while the self-attention analysis later distinguishes positions.Source: §3.2 · pp. 6–7

3.3 Parameter Initialization

  • Parameter matrices use coordinate standard deviation d^-γ, with γ > 1/2 defining the small-initialization regime analyzed for the relative scale of signature contributions.This regime is related to the condensed/nonlinear training regime studied in prior work (Luo et al., 2021; Zhou et al., 2022).Source: §3.3 · p. 8

3.4 Main Result: Context Staircase

  • Under small initialization, embeddings first reflect low-order signatures—especially label statistics—then increasingly reflect higher-order context; in addition-task embeddings, an odd–even structure emerges later.Higher-order contributions are initially suppressed because they involve more embedding factors; later they can jointly shape geometry with lower-order signatures, with the highest accessible order most prominent under the analyzed conditions.Source: §3.4 · pp. 8–9

4 Signature-Aligned Dynamics in Feed-Forward Models

This section analyzes signature-aligned token-embedding dynamics in a controlled feed-forward model, using gradient flow to explain how small initialization shapes their progression from zero-order label signatures toward higher-order signatures. It also examines how signature orders act together over training and connects this progression to spectral refinement, task-dependent benefits, and signature accessibility.

4.2 Empirical Signature Alignment

  • Early token embeddings match their zero-order label distributions, and numerical embedding geometries reproduce the corresponding signature geometries across arithmetic tasks.The distribution match holds across uniform, normal, Poisson, and exponential targets; addition and subtraction show ordered geometry, while multiplication forms clusters.Source: §4.2 · pp. 10–11, Figure 2 · p. 12
  • The zero-order signature explains embedding geometry almost perfectly early in training, but its explanatory power declines as first-order signatures become stronger later.In modular addition, the zero-order signature cannot distinguish tokens, and later embeddings instead form a modulo-19 ring; addition develops an odd-even separation.Source: §4.2 · p. 13, Figure 3 · p. 14

4.3 Gradient-Flow Explanation

  • The label-driven embedding gradient explicitly includes label-signature contributions from order 0 through the largest accessible order.With a first-order softmax expansion, the prediction-induced term includes signatures up to min{L −1, K −1}; retaining higher softmax orders can introduce still higher orders, up to L −1.Source: §4.3 · pp. 15–16

4.4 Order Hierarchy under Small Initialization

  • With small initialization, higher-order terms are suppressed relative to the zero-order label signature at the start, then become comparable as embedding scales grow.In the stated late-stage regime, accessible orders share the same leading scale, with the highest-order contribution largest.Source: §4.4 · p. 17
  • At small initialization, signature contributions follow m = 0 > m = 1 > m = 2, but later share a parameter scale, with the highest order potentially strongest.Initially, each increase in signature order adds a factor of the small embedding scale; later, the common scale removes this suppression and the highest accessible order has the largest combinatorial prefactor when K ≫ L.Source: §4.4 · p. 18

4.5 Relations across Signature Orders from a Fourier Frequency Perspective

  • Higher-order signatures contain lower-order signatures in their zero-frequency projections, and embeddings progress from matching that shared structure to expressing nonzero-frequency components.Changing only the random seed can also produce substantially different embedding geometries, explained by alignment with different signature frequency components.Source: §4.5 · p. 19

4.6 A Possible Benefit of Context Staircase.

  • Context Staircase imposes an ordered rather than immediately task-optimal learning process: it may delay learning when low-order signatures lack task-relevant information, as in modular addition.Its possible benefit is an inductive bias that introduces simpler relations before more context-dependent ones; whether it accelerates or delays optimization depends on which signature orders contain the needed information.Source: §4.6 · pp. 20–21

4.7 Activation Order Controls the Accessible Signature Order

  • Learning succeeds only when activation order reaches sequence length; below that threshold, number-token embeddings collapse into the same direction.The tested modular-addition tasks had identical lower-order signatures across number tokens, so only the highest-order, full-context signature distinguished them.Source: §4.7 · p. 21, Figure 5 · p. 22

5 Attention-Modulated Signature Dynamics

The section shows that self-attention preserves signature-based embedding dynamics while making them position-specific and attention-weighted. Empirically, attention-only models still progress from early zero-order to later first-order structure, and attention scores shape how quickly embeddings evolve.

5.2 Gradient-flow Analysis

  • Theorem 3 decomposes last-token label-driven gradients into value, key, and query terms: signatures are position-specific and attention-weighted, with explicit attention scores leaving signatures at most second-order.Further expanding the attention-score terms could produce higher-order signatures, but the analysis retains the scores explicitly to characterize attention’s direct effect.Source: §5.2 · pp. 23–25

5.3 Empirical Consequences of Attention Modulation

  • In attention-only models, Fadd and Fmod embeddings shift from early zero-order label-signature dominance to visible first-order structure later, as in the feed-forward analysis.This shows progressive signature alignment is not specific to feed-forward models.Source: §5.3 · p. 25, Figure 6 · p. 26
  • Anchor tokens receive higher attention and change faster, while noise-token attention decays, leaving their gradients negligible and their embeddings evolving more slowly.The attention and embedding-norm dynamics show that attention controls the rate and magnitude of embedding evolution.Source: §5.3 · pp. 25–26, Figure 6 · p. 26

6 Path-Specific Signatures in Transformer Training

The section explains how direct, attention, and feed-forward paths contribute distinct signatures to Transformer embedding updates, then tests whether embedding geometry and attention-path representations reflect the predicted progression during real training.

6 Path-Specific Signatures in Transformer Training

  • A full Transformer’s embedding gradient is the additive sum of direct, attention, and FFN path contributions, linking the modular analyses to full-model embedding dynamics.Source: §6 · pp. 26–27
  • The direct embedding-readout path updates each token embedding according to its next-token signature: the distribution of tokens that follow it.Source: §6 · p. 28
  • Early in training, the FFN path induces only zero-order next-token signatures; as attention outputs grow, its context-bearing input can induce higher-order signatures.The FFN receives a single-position residual stream, initially dominated by the direct term, and later a weighted combination of preceding-token embeddings.Source: §6 · p. 28
  • Early attention paths contribute attention-weighted zero-order token–label signatures, while later key- and query-dependent terms introduce position-specific and higher-order signatures.Tokens assigned greater attention mass exert stronger influence on the corresponding embedding update.Source: §6 · pp. 28–29

6.1 Experimental Validation

  • Embedding geometry is initially dominated by next-token signatures, whose similarity correlation peaks early then falls to a stable nonzero level, while a rank-one first-order probe stays low-error and generalizes to held-out tokens.The probe is trained on one subset of anchor tokens and evaluated on unseen tokens.Source: §6.1 · p. 30, Figure 7 · p. 31
  • Attention-path representations show clear early-stage correlation with the corresponding attention-weighted signatures, supporting the path-specific prediction.Figure 7C reports layer-wise correlations across training steps.Source: §6.1 · p. 30, Figure 7 · p. 31

7 The Role of Signatures in Model Learning

Signature structure can support task learning but is not sufficient without an architecture able to exploit it. The section also shows that zero-order signatures can support a task rule and encode semantic organization.

7.1 Relation between Signature and Learning

  • In modular addition, FFN embeddings aligned with first-order signatures accompany rapid task learning, but attention-only embeddings show similar structure while the model fails to solve the task.The attention-only model is not expressive enough to exploit the learned signature structure.Source: §7.1 · pp. 31–32, Figure 8 · p. 32
  • A fixed first-order-signature embedding converges substantially faster than trainable embeddings, while fixed random embeddings also outperform trainable embeddings early in modular addition.All number tokens share the same zeroth-order label signature, so early updates provide little token distinction; approximately orthogonal random embeddings preserve distinctions from the start.Source: §7.1 · p. 32
  • With N = 20, moduli M = 4, 5, 10, and 20 require thousands of epochs to reach 95% accuracy, whereas other tested moduli typically take only a few hundred.When M divides N, zeroth-order signatures are uniformly similar and do not distinguish tokens; otherwise, they distinguish residue classes. The onset of discriminative signature information predicts learning speed.Source: §7.1 · p. 33, Figure 9 · p. 33

7.2 An Example of Task Solving via Signatures

  • In the contextual min–max task, numerical zero-order signatures create candidate peaks at x1 and x2, while the anchor signature selects the larger candidate for a and the smaller for b.Their additive superposition suffices to support the required prediction rule when the architecture can access it; the paper does not claim every trained model implements this exact score.Source: §7.2 · pp. 34–35

7.3 Signatures Encode Semantic Structure

  • Signatures from a 36B-token training corpus trace numerical order, organize weekdays cyclically, and cluster tokens from shared semantic categories in projected space.These findings suggest that semantic geometries in learned embeddings may originate in token-conditioned corpus statistics that progressively enter embedding updates.Source: §7.3 · pp. 35–36

8 Conclusion … A.1.1 FFN Architecture

Token embeddings progressively align with token-conditioned signatures: simple label distributions shape early geometry, while higher-order contextual signatures become jointly visible later. Gradient-flow analyses explain how initialization and architecture shape this evolution, and task usefulness depends on whether the model can exploit the encoded structure.

8 Conclusion

  • Early embedding geometries align with simple token-conditioned label distributions, while later training reveals multiple higher-order contextual signatures, with the highest accessible order strongest under late-stage conditions.The higher-order signatures do not replace one another in a strict sequence.Source: §8 · pp. 36–37
  • Figure 11 projects signatures for numerical tokens in numerical order, weekdays in cyclic order, and tokens from semantic categories including colors, family roles, countries, and programming languages.Source: Figure 11 · p. 37
  • Low-order structure corresponds to zero-frequency projections of higher-order signatures; task-specific nonzero-frequency components emerge later, and random-seed geometries reflect different spectral selections within a shared hierarchy.Source: §8 · p. 37
  • In FFNs, small initialization suppresses higher-order signature contributions in embedding gradients; in self-attention, attention scores modulate position-specific signatures and learned routing determines which token statistics most affect embeddings.Source: §8 · p. 37
  • In real language-model training, early embedding geometry tracks next-token signatures associated with direct and FFN paths, while attention-path representations track attention-weighted signatures across layers; these correlations later decay as additional structures enter the model.Source: §8 · p. 37
  • Signature-aligned geometry alone does not ensure task success: it can support prediction and accelerate optimization when the architecture can exploit it, but structured embeddings may emerge without successful learning otherwise.Source: §8 · pp. 37–38

Appendix A. Experimental Configurations

  • The appendix gives configurations for controlled FFN, attention-only, signature-based embedding, and real language-model experiments, with common settings applying unless otherwise specified.Source: Appendix A · p. 39

A.1.1 FFN Architecture

  • Controlled FFN models use d = 128 by default without biases, normalization, residual connections, dropout, or positional embeddings; Figures 1–3 and 7 use ReLU, while Figure 4 uses the monomial activation σ(z) = z⊙K.Source: §A.1.1 · p. 39

A.1.2 Prescribed Label-Distribution Experiment … B.2 Supplementary of Remark 1

The appendix details controlled label-distribution and arithmetic experiments, signature-projection and embedding analyses, attention and language-model setups, and theoretical derivations. Together, these methods and derivations specify how first-order and higher-order signatures are examined in the experiments and gradient flow.

A.1.2 Prescribed Label-Distribution Experiment; A.1.3 Arithmetic Tasks; A.1.4 Addition and Modular-Addition Dynamics

  • Task 1 uses anchor a = 100 and X = {0, . . . , 50}; labels are fixed after one-time sampling from four distributions, while addition and modular-addition models train 1200 epochs (Figure 2: epochs 100, 1100).Source: §A.1.2 · p. 39, §A.1.4 · p. 41

A.1.5 Optimized First-Order Signature Projection; A.1.6 Fourier Analysis of the First-Order Signature

  • The first-order signature analysis optimizes a probe vector to match pairwise signature similarities, then measures embedding agreement with Fourier cosine components across frequencies.The probe is evaluated with five-fold cross-validation; cosine components are analyzed because the zero-frequency sine basis is identically zero.Source: §A.1.5 · p. 41, §A.1.6 · p. 42

A.1.7 Activation-Order Sweep; A.2.1 Attention Architecture; A.2.2 Attention Arithmetic Experiments

  • The activation-order sweep varies sequence length and monomial degree; attention experiments specify a causal architecture and train arithmetic models on all ordered pairs.The activation sweep uses lengths 2–4 and degrees 1–6; the attention architecture has no residual connection, normalization, or FFN module.Source: §A.1.7 · p. 42, §A.2.1 · p. 42, §A.2.2 · p. 42

A.2.3 Anchor–Noise Parity Task

  • Figure 5K reports, over the full dataset, average attention mass assigned to the anchor token and to the noise tokens; for a single-anchor sequence, anchor mass is attention weight assigned to the anchor position.The training set is generated once with 10000 samples, and the model is trained for 200 epochs.Source: §A.2.3 · p. 43

A.3 Signature-Based Embedding Experiments; A.3.1 Fixed First-Order Signature Embedding; A.3.2 Baselines

  • The fixed first-order signature embedding uses a random projection with fixed random components and trainable scale, alongside jointly trainable and fixed-random embedding baselines.The empirical first-order label signature is computed for each token before constructing the fixed signature embedding.Source: §A.3.1 · pp. 43–44, §A.3.2 · p. 44

A.3.3 FFN and Attention Comparison in Figure 7; A.4 LLMs

  • Figure 7A–C presents Fmod experiments with L = 2 and N = 40, training models for 1000 epochs under identical optimization settings.Language-model experiments use a 0.7B dense decoder-only Transformer trained with next-token prediction on a bilingual Chinese–English corpus containing 36B tokens. The model has 24 layers and approximately 680M non-embedding parameters.Source: §A.3.3 · p. 44, §A.4 · p. 44

B.1 Proof of Theorem 1,2

  • The proof derives symmetric context-index kernels for the gradient terms and uses a first-order softmax expansion around the origin when logits are small.The expansion is used for early training, where the passages state that logits are small.Source: §B.1 · p. 46

B.2 Supplementary of Remark 1

  • The supplementary analysis expands the softmax contribution to the embedding gradient into uniform-prediction and linearized-logit parts, showing that the expansion induces higher-order signatures.It treats the higher-order softmax remainder separately from the previously analyzed uniform-prediction contribution.Source: §B.2 · pp. 47–49

B.3 Proof of Proposition 1

At initialization, every fixed higher-order label-signature contraction is asymptotically smaller than the zero-order contraction. The proof also establishes normalization, marginal consistency, and norm monotonicity for the signatures.

B.3 Proof of Proposition 1

  • The signature norms are monotone across orders under marginalization, and the same normalization, consistency, and norm properties hold for co-occurrence signatures without the label index.The proof obtains the norm inequality using nonnegativity, applies marginal consistency, and iterates over m; the co-occurrence case is identical.Source: §B.3 · p. 50
  • A label signature is a joint distribution over the label and tokens at uniformly sampled distinct context positions, so it is nonnegative, normalized, and marginally consistent across orders.Deleting one context coordinate leaves a uniformly sampled tuple of the lower order, and summing over the deleted token yields that order’s marginal.Source: §B.3 · p. 50
  • Every fixed higher-order label-signature contraction is asymptotically smaller than the zero-order contraction at initialization.For order m, the lowest-degree term is k = m + 1; higher-degree terms have additional factors of scale d^-γ.Source: §B.3 · pp. 50–51

B.4 Proof of Proposition 2

The proof shows that after the embedding scale crosses a common threshold, the highest polynomial-degree term dominates within each signature order. For sufficiently large degree, the highest signature order then has the largest contribution.

B.4 Proof of Proposition 2

  • Once sE(t) ≥ S∗, degree K dominates every lower-degree term within each fixed signature order m.A common threshold works across the finitely many orders m = 0, . . . , L −1.Source: §B.4 · p. 52
  • At late stages, all signature orders scale as sU(t)sE(t)^(K−1), so their hierarchy is no longer determined by different powers of sE(t).This follows from the dominant degree-K terms and uniform non-degeneracy of normalized contractions and signatures.Source: §B.4 · p. 52
  • The order-dependent prefactors are governed exponentially in K by Bm(K), identified as a Stirling number of the second kind.Under uniform non-degeneracy, sufficiently large K makes the highest-order prefactor the largest.Source: §B.4 · pp. 52–53
  • For sE(t) ≥ S∗ and sufficiently large K, the order-(L −1) term is largest among all signature contributions.Its prefactor is largest among the same-scale signature-order contributions.Source: §B.4 · p. 53

B.5 Proof of Theorem 3

The proof decomposes the attention-gradient update into value and score paths and expresses their contributions using label and context signatures. The value path gives an attention-reweighted label-signature update, while the score path splits into key- and query-score terms.

Value path

  • The value-path contribution is a label-signature update reweighted by the attention mass assigned to token α.The proof identifies this as its first signature term.Source: §B.5 · p. 54

Score path

  • The score-path gradient separates into key-score and query-score contributions, with α affecting the query when it is the last token and the key when it occurs at position u.The proof derives the score-path contribution from the softmax derivative before naming the two terms.Source: §B.5 · pp. 54–55

Key-score contribution

  • Conditioned on α, the key-score term depends on the joint label–last-token distribution and is rewritten using label–last-token and label–last-token–context signatures.The proof also defines attention-weighted quantities for these signatures.Source: §B.5 · p. 55

Query-score contribution

  • The label-driven query-score term is rewritten using label-and-context distributions conditional on xL = α, along with corresponding attention-weight moments.The defined signatures include one context token and pairs of context tokens.Source: §B.5 · p. 56

Appendix C. Multi-stage embedding evolution in length-three sequences.

In length-three tasks, anchor embeddings move from a transient zeroth-order geometry to a late-stage geometry jointly shaped by first- and second-order signatures, with the second-order signature fitting more strongly.

Appendix C. Multi-stage embedding evolution in length-three sequences.

  • After initialization-induced suppression lifts, first- and second-order signatures can contribute jointly at the same leading parameter scale, with the second-order contribution expected to be largest.The length-three task exposes these higher-order contributions together.Source: Appendix C · p. 57
  • Task 4 uses anchor tokens and a contextual token to predict the sum of their integer values modulo |X|.The input space is A × A × X, with target y = F(a1, a2, x) = (a1 + a2 + x) mod |X|.Source: Appendix C · p. 57
  • All anchor tokens share identical zeroth-order label signatures, but first- and second-order token–label statistics distinguish anchor identities.The anchors and contextual token are drawn from disjoint token ranges.Source: Appendix C · p. 57
  • Anchor embeddings evolve from nearly uniform similarities and a compact PCA cluster, through a banded pattern and smooth curve, to more complex late-stage geometry.The late-stage geometry is consistent with joint first- and second-order signature influence, rather than one order replacing the other.Source: Appendix C · p. 57
  • Correlations with both first- and second-order signatures rapidly approach one and remain high, while the zeroth-order correlation turns strongly negative before returning toward zero.In the late stage, the second-order signature provides the stronger fit, consistent with joint higher-order contributions and the largest contribution coming from the highest accessible order.Source: Appendix C · p. 57

Appendix D. Different Seeds, Different Embedding

With the dataset, architecture, and training configuration fixed, changing the random seed can produce different final embedding geometries in addition and modular-addition tasks. These differences are associated with embeddings aligning to different frequency components of the same higher-order signature.

Appendix D. Different Seeds, Different Embedding

  • The analysis compares each seed’s final pairwise embedding similarities with similarity matrices induced by Fourier components of its corresponding signature.The reported correlations concern the off-diagonal embedding-similarity entries and signature components indexed by frequency.Source: Appendix D · pp. 58–59, Figure 13 · p. 59
  • With data, architecture, and training fixed, different seeds produce distinct final geometries in addition and modular-addition, with embeddings aligning to different frequencies of the same higher-order signature.The frequency selected varies across seeds, and the learned embedding PCA geometry closely matches the projection of the selected signature component.Source: Appendix D · pp. 57–59, Figure 14 · p. 60, Figure 15 · p. 61

Related papers

Loading 2608.30315v1…