Source-linked AI summary
Continuity and Isolation Lead to Doubts or Dilemmas in Large Language Models
Hector Pasten, Felipe Urrutia, Hector Jimenez, Cristian B. Calderon, Cristóbal Rojas, Alexander Kozachinskiy
TL;DR
The paper asks what sequences decoder-only Transformers with compact positional encoding can learn, given the importance of understanding Transformer capabilities. It proves continuity and isolation properties and examines their practical occurrence, concluding that Transformers face doubts, dilemmas, or both when learning even simple sequences.
Problem
The paper addresses the need for a theoretical understanding of what Transformers can compute, alongside empirical study of LLM task performance.
Source: §1 · p. 1Method
The authors mathematically prove continuity and isolation for decoder-only Transformers with compact positional encoding and design experiments to examine these limitations in practice.
Source: §1.1 · p. 2, §3.1 · p. 6Results
Continuity and isolation imply that Transformers face inevitable dilemmas, doubts, or both; experiments report varying sensitivity across models, including 21% for Meta-llama-3-8B-Instruct.
Source: §3.2 · p. 7, §5 · p. 9Takeaways & Limitations
Under the paper’s assumptions, a Transformer cannot avoid both continuity and isolation, so it must face either doubts or dilemmas, or both.
Source: §5 · p. 9The results require compact positional encodings; they do not cover absolute positional encoding, log n-scaling, or layer normalization with ε = 0.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Understanding how Transformers work and how they process information is key to the theoretical and empirical advancement of these machines. In this work, we demonstrate the existence of two phenomena in Transformers, namely isolation and continuity. Both of these phenomena hinder Transformers to learn even simple pattern sequences. Isolation expresses that any learnable sequence must be isolated from another learnable sequence, and hence some sequences cannot be learned by a single Transformer at the same time. Continuity entails that an attractor basin forms around a learned sequence, such that any sequence falling in that basin will collapse towards the learned sequence. Here, we mathematically prove these phenomena emerge in all Transformers that use compact positional encoding, and design rigorous experiments, demonstrating that the theoretical limitations we shed light on occur on the practical scale.
1 Introduction
The paper studies limits on what decoder-only Transformers with compact positional encoding can learn, extending theoretical analysis to an architecture used in modern LLMs. It establishes continuity and isolation, derives consequences for learnable sequences, and tests whether the predicted limitations appear in practice.
1.1 Our contribution
- Isolation means every eventually learned infinite sequence has a positive-radius neighborhood containing no other distinct eventually learned sequence.Distance is measured by the asymptotic frequency of differing positions.Source: §1.1 · p. 2
- Continuity means small prompt changes cannot change the output distribution beyond a model-dependent threshold, provided the final token remains unchanged.This yields representational collapse: nearby prompts share the same top-probability next token under the stated prediction assumptions.Source: §1.1 · pp. 2–3
- Isolation rules out jointly learning some pairs, including the all-0 sequence and the increasing-spacing pattern, despite their differences having frequency tending to zero.The paper also concludes that no single decoder-only CPE Transformer can eventually learn the entire family of periodic sequences, although any finite subfamily can be learned.Source: §1.1 · p. 3
- For an eventually learned sequence, isolation implies that sufficiently long prefixes overwhelm the Transformer: appending a sufficiently small fraction of tokens leaves its output unchanged.The paper connects this consequence to prior work on detecting overwhelmed strings [15].Source: §1.1 · pp. 3–4
1.2 Related work
- Prior continuity results addressed encoder-only Transformers; extending the analysis to decoder-only models requires controlling the greater influence of earlier tokens under causal masking.The paper presents this control as its main technical contribution and states that its results require continuous value, attention, and activation functions, but not Lipschitz continuity.Source: §1.2 · p. 4
1.3 Experiments
- Experiments examine continuity on zero-sequence and Python syntax-error tasks and isolation through periodic-sequence generation, testing whether theoretical limitations occur in modern models.The paper says the experiments provide evidence of practical relevance and compare how architectural differences affect limitation severity.Source: §1.3 · p. 4
2 Preliminaries
The preliminaries define the Transformer and decoder-only attention operations used in the paper. They emphasize compact positional encoding and the prefix dependence imposed by causal attention.
2 Preliminaries
- A decoder-only attention layer computes weights and values from input vectors and positional encodings, forms attention vectors, then applies an activation function.The layer’s output at position j depends only on inputs through position j, making it prefix-monotone.Source: §2 · p. 5
- Compactness requires positional encodings to remain within a compact set, while compact input embeddings and a constant number of attention heads are also accommodated.The paper uses the infinity norm by convention, while stating that its results hold for other norms in finite dimensions.Source: §2 · p. 5
- A Transformer maps finite token sequences to probability distributions over tokens, using embeddings, attention layers, and an output function.The model definition specifies a decoder-only Transformer with a finite alphabet and a fixed number of layers.Source: §2 · p. 5
3 Continuity
Decoder-only compact Transformers are continuous: sufficiently small relative prompt changes keep output distributions close, supporting the same next-token prediction for similar prompts. Experiments find this pattern around all-zero prompts and in syntax verification, with sensitivity varying by perturbation location, model, and attention variant.
3 Continuity
- For every ε > 0, sufficiently small relative Hamming distance between equal-length prompts ending in the same token keeps a compact decoder-only Transformer's output distributions within ε.The next-token propagation principle predicts the same next token when one prompt's prediction is sufficiently certain.Source: §3 · p. 6
3.1 Empirical support for continuity: Zero fundamental sequence
- At γ = 0.01, no tested model diverged from the all-zero prompt; at γ = 0.05, NTSγ was zero for all but two models.Perturbations affected sensitivity most at the sequence end and least at the beginning; log-length scaled attention increased sensitivity relative to standard softmax.Source: §3.1 · p. 6, Figure 2 · p. 6
3.2 Empirical support for continuity: Code Syntax Verification
- In syntax verification, most examples showed no sensitivity, but rates varied by model: Meta-llama-3-8B-Instruct was sensitive on 21% of examples, while phi-4 gave the same output for all cases.The test pairs differed by one token in a Python syntax exercise, and sensitivity was measured by comparing the target-token probability under each prompt.Source: §3.2 · p. 7
4 Isolation
Isolation means a sequence learned by a decoder-only compact Transformer has a positive-radius neighborhood containing no other learnable sequence that differs from it infinitely often. The section also shows that periodic-pattern generation has a critical period, which increases with repetitions.
4 Isolation
- For any eventually learned sequence α, some δ > 0 excludes every sequence β within relative Hamming distance δ that differs from α infinitely often from being eventually learned by the same Transformer.Sequences differing in only finitely many positions have the same eventual learnability status.Source: §4 · pp. 7–8
- No decoder-only compact Transformer can eventually learn all periodic sequences, although a single Transformer can learn any finite family whose members are pairwise a positive relative Hamming distance apart.The same isolation result rules out learning multiple sequences that differ infinitely often but have relative Hamming distance zero.Source: §4 · p. 8
4.1 Empirical support for isolation: Periodic Pattern Generation
- In Llama-2-7b-hf, the critical period beyond which periodic sequences are not reproduced correctly increases with repetitions, and certainty dips around that boundary.The evaluation varies periods from 2 to 40 and uses 1, 4, or 10 repetitions; near the boundary, generation switches from the expected 1 to 0.Source: §4.1 · pp. 8–9, Figure 4 · p. 9
5 Conclusion – Doubts and Dilemmas
Continuity and isolation force Transformers either to leave some simple sequences unlearned or to give up certainty in next-token prediction. The results require compact positional encoding, while the theoretical scope of growing-length chain-of-thought remains open.
5 Conclusion – Doubts and Dilemmas
- Continuity and isolation imply that Transformers face either dilemmas or doubts, or both, even among simple sequences.Avoiding dilemmas requires giving up the no-doubts assumption: the next token need not be predicted with a probability margin.Source: §5 · p. 9
- The results require compact positional encoding, which includes sinusoidal and rotary encodings but excludes absolute positional encoding and log n-scaling.Layer norm with ε > 0 is within scope; ε = 0 is not, because layer norm is then discontinuous and may be undefined.Source: §5 · p. 9
- A constant number of chain-of-thought iterations is covered by the model, but the argument does not extend when iteration count grows with input length.The theoretical power of softmax Transformers with compact positional encoding in that growing-iteration regime remains open.Source: §5 · pp. 9–10
A.1 Proof of Theorem 1
The proof establishes continuity by showing that nearby token sequences with a shared final token produce nearby final representations after any fixed number of attention layers. It propagates both global similarity and last-position similarity through the layers.
A.1 Proof of Theorem 1
- The proof compares sequences with the same last token and small relative Hamming distance, then bounds how close their final representations remain.Inputs are embedded using token and position, and the embeddings lie in a fixed compact set independent of sequence length.Source: §A.1 · p. 11
- The induction maintains both global similarity across sequence positions and closeness of the final vectors, since final-position closeness alone is insufficient.Global similarity requires that all but a small fraction of positions have vectors within a small distance.Source: §A.1 · p. 11
- A single attention layer preserves these similarities because bounded continuous value, weighting, and activation functions keep representations in compact sets.Convex combinations of value vectors remain in a bounded ball, and continuity of the activation maps that ball to a compact set.Source: §A.1 · pp. 11–12
A.1.1 Proof of Lemma 1
Lemma 1 follows by first proving that a layer preserves closeness at the final position, then extending that bound to most output positions. The estimates use compactness and uniform continuity to control attention weights and value vectors.
A.1.1 Proof of Lemma 1
- Lemma 2 supplies final-position closeness when input sequences are globally similar and their last vectors are close.The proof places inputs, values, positional encodings, and attention vectors in a common bounded ball.Source: §A.1.1 · p. 12
- Only a small fraction of input positions can be bad, so for most sufficiently late positions the prefixes remain globally similar and Lemma 2 bounds their outputs.Positions before εn/2 are handled separately; the remaining exceptions are limited by the fraction of bad inputs.Source: §A.1.1 · p. 12
- Uniform continuity on compact sets makes changes in weights and value vectors small when inputs are close, while bounded weights control contributions from exceptional positions.The denominator is bounded below by c^2n^2, and the numerator is o(1)n^2 as δ → 0.Source: §A.1.1 · pp. 12–13
- The theorem also holds for encoders because the proportion of exceptions does not change across iterations.The argument is simpler than for decoder-only Transformers.Source: §A.1.1 · p. 13
A.2 Proof of Proposition 1
A sequence learned with a positive prediction margin remains learned after finitely many token changes. Continuity transfers the prediction margin to the modified sequence for sufficiently long prefixes.
A.2 Proof of Proposition 1
- Finite changes to an eventually learned sequence preserve eventual learnability, with the prediction margin reduced from ε to ε/2.For sufficiently long prefixes, the original and modified sequences share their last token and have relative Hamming distance at most δ, so their output distributions are ε/10-close in ℓ∞ norm.Source: §A.2 · p. 14
A.3 Proof of Theorem 2
The proof uses continuity to show that a sequence sufficiently close to an eventually learned sequence, but differing infinitely often, cannot also be eventually learned. This yields the corollary that no decoder-only compact Transformer eventually learns all periodic sequences.
A.3 Proof of Theorem 2
- Continuity bounds output-distribution changes for close equal-length prefixes with the same last token, providing the key tool for the contradiction.For any ε, Theorem 1 supplies a δ such that relative Hamming distance at most δ makes outputs ε/3-close in the ℓ∞-norm.Source: §A.3 · p. 14
- A long prefix pair with bounded relative Hamming distance contains a sufficiently long matching final token, after which the first divergence violates eventual learnability.The proof finds ℓ with αℓ = βℓ and prefix distance below δ, then takes the first subsequent divergence and obtains a contradiction to β’s eventual learnability.Source: §A.3 · p. 14
A.4 Proofs of Corollary 2
- If one decoder-only compact Transformer learned every periodic sequence, it would learn the all-0 sequence and a nearby periodic sequence with infinitely many 1s, contradicting Theorem 2.Choosing a period k ≥ 1/δ gives relative Hamming distance 1/k ≤ δ.Source: §A.4 · p. 15
B.1 Visualization of Decoder-only Architecture
These visualizations connect decoder-only Transformer structure and continuity to experiments on input sensitivity. The experiments find stronger effects from later perturbations and greater next-token variability with ssmax than with standard softmax.
B.1 Visualization of Decoder-only Architecture
- The decoder-only schematic traces embedded tokens through causal attention layers to a final vocabulary distribution, with each position depending only on its prefix.The final layer produces y_n, which a projection maps to a probability distribution.Source: §B.1 · p. 15, Figure 5 · p. 15
B.2 Visualization of Continuity
- The continuity illustration contrasts perturbations inside and outside a relative-Hamming threshold: within it, the next-token distributions remain close.The threshold δ bounds the input perturbation, while ε bounds the output-distribution difference.Source: Figure 6 · p. 15, §B.2 · p. 16
B.3 Effect of Divergent Positions
- Across models, perturbations near the input’s end affect next-token predictions more than perturbations near its beginning or middle.The experiment varies perturbation positions using eight positional biases.Source: §B.3 · p. 16, Figure 7 · p. 16
B.4 Boosting sensitivity
- Replacing standard softmax with ssmax significantly increases next-token variability under small input changes.Figure 8 compares the number of sequences whose next token differs from zero against the proportion of differing symbols, with standard softmax on the left and ssmax on the right.Source: §B.4 · p. 17, Figure 8 · p. 17
B.5 Visualization of Code Syntax Verification
The appendix tests sensitivity to small code changes and evaluates periodic-pattern generation across models. Syntax errors remain common even for instruct-tuned and reasoning models, while periodic completion degrades beyond model-specific period thresholds.
B.5 Visualization of Code Syntax Verification
- The syntax-verification task gives two annotated Python examples before a target function and asks whether the target has a syntax error.The two prompt versions differ by only a few tokens, allowing sensitivity to small syntactic changes to be assessed.Source: §B.5 · p. 17, Figure 9 · p. 17
- Syntax-verification error rates remain above 15% for instruct models and above 80% for base models, although instruct models perform significantly better.The task compares prompts differing by a few tokens in whether the target Python function contains a subtle syntax error; the figure uses a simpler function dataset than the main experiment.Source: §B.5 · pp. 17–18, Figure 10 · p. 18
B.6 Syntax Code Verification on Reasoning Models
- Reasoning models o3-mini and o4-mini still show error rates of at least 20% on the simple syntax-verification prompts.The authors suggest these results indicate that chain-of-thought cannot always escape continuity at practical scales.Source: §B.6 · p. 18
B.7 Effect of Different Models for Periodic Generation
- Models generally extrapolate periodic patterns perfectly or nearly perfectly for p ≤ 10, then degrade and eventually fail beyond model-specific critical periods.The reported limitation appears across models, with gemma-2b and gemma-2-2b noted as exceptions.Source: §B.7 · p. 18, Figure 12 · p. 20
C Prompt formatting
- The prompt-format appendix keeps the instruction structure consistent across models while adapting special tokens, including for reasoning models.The same structure is used to support comparable results.Source: C Prompt formatting · p. 19