PaperlayerView exact revision on arXiv

Source-linked AI summary

When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodings

Christopher Schröder, Lukas Gienapp, Ferdinand Schlatt, Martin Potthast, Gerhard Heyer

arXiv:2608.03994v1cs.CL

TL;DR

ALiBi positional biases can underflow floating-point precision, partially blinding attention heads, while the downstream impact of this previously unreported failure remained unclear. The paper analyzes the mechanism, verifies it in pretrained models, and tests mitigations, finding larger effects on retrieval than standard benchmarks and no universally dominant strategy.

  • Problem

    ALiBi’s previously unreported attention-underflow failure and its impact on downstream effectiveness remain insufficiently characterized.

  • Method

    The paper analytically characterizes underflow, evaluates pretrained ALiBi models, trains controlled decoder models, and tests isolated and combined mitigation strategies.

  • Results

    Standard benchmarks differed only 1.6 to 3.4 pp, while retrieval effects were larger; clamping plus log-scaled distances improved out-of-context passkey retrieval nearly ten-fold (0.79 vs. 0.08 AUC), but no strategy dominated every task.

  • Takeaways & Limitations

    ALiBi remains a strong needle-in-a-haystack baseline, but its numerical failure can substantially impair retrieval despite minor effects on standard decoder benchmarks.

  • Takeaways & Limitations

    Because experiments used 148M-parameter decoders with fixed architectures and corpora, the findings may not translate to larger models or other experimental settings.

Abstract

from arXiv · show

We identify a previously overlooked failure mode of ALiBi positional encoding: its linear bias scaling underflows floating-point precision, which zeroes out a large fraction of attention weights and renders the affected attention heads partially blind. We analyze this failure mode, characterize its impact, and examine four mitigation strategies. We further demonstrate its occurrence in state-of-the-art pretrained models based on ALiBi. Comprehensive pretraining experiments with 148M-parameter decoder models help us to disentangle its effects from out-of-context degradation. We find that ALiBi's failure mode can substantially impair token retrieval while having only a minor effect on standard decoder benchmarks. We propose four training-time mitigation strategies and evaluate them individually and in combinations, finding that log-scaled distances yield the most consistent improvements in passkey retrieval. Despite this problem, default ALiBi slopes remain a surprisingly strong baseline, particularly for needle-in-a-haystack retrieval. Based on these findings we provide concrete recommendations on how to train models with ALiBi.

1 Introduction

ALiBi’s unbounded linear bias can cause attention weights to underflow in finite-precision arithmetic, making affected heads blind to sufficiently distant token interactions. The paper investigates this failure in pretrained and newly trained models and evaluates training-time mitigations.

  • Background: ALiBi encodes relative position with an additive distance-dependent bias and is computationally cheap, parameter-free, and memory-efficient.Its proposed advantage is straightforward extrapolation beyond the training context window.
  • Failure mode: Unbounded linear bias scaling can make attention weights underflow, zeroing affected scores beyond a distance threshold and rendering heads partially blind.The failure arises from finite floating-point arithmetic rather than ALiBi’s formal definition alone.
  • Motivation: This failure mode counteracts ALiBi’s promise of length extrapolation, and its downstream impact had not previously been established.The paper presents the problem as previously unreported and systematically investigates its consequences.
  • Contributions: The study assesses pretrained ALiBi models, trains decoder models to characterize triggering conditions, and evaluates several training-level mitigation strategies.The experiments are organized across pretrained-model assessment, extensive decoder training, and mitigation analysis.

2 Related Work

This section situates ALiBi among positional encodings for Transformers, contrasting additive distance biases with sinusoidal and rotary approaches. Recent numerical and long-context failure modes motivate renewed attention to ALiBi and other alternatives.

  • Transformer positional encodings: Self-attention processes tokens in parallel without sequential order, so positional encoding injects word-order information into Transformer models.The paper frames positional encoding as essential background for the approaches reviewed.
  • Absolute positional encodings: Fixed sinusoidal encodings add positional information to token embeddings, while learned absolute encodings were reported similarly effective but less extrapolative.Sinusoidal embeddings can extrapolate to some degree beyond the training context length.
  • ALiBi: ALiBi adds a distance-growing bias to attention logits, requiring little compute and enabling attention beyond the training context length.Its additive bias depends on the distance between query and key.
  • Rotary positional encodings: RoPE encodes position through position-dependent query-key rotations, carries relative information, and became the default after outperforming ALiBi in later comparisons.The paper returns to the ALiBi side of that comparison.
  • Failure modes and alternatives: Recent work identifies RoPE failures involving weakened low-frequency signals, degraded position and token discrimination, and compromised relative positions under bfloat16 precision.These shortcomings motivate reconsidering alternatives, with ALiBi described as the cheapest available option.

3 Numerical Failure in ALiBi

ALiBi’s linearly distance-scaled bias eventually pushes softmax exponentials below floating-point precision, causing attention heads to become partially or fully blind. The section characterizes the resulting positional blindness and weight redistribution, then introduces four non-exclusive mitigation strategies.

  • Mechanism: ALiBi’s negative bias grows with token distance and eventually drives softmax exponentials below the floating-point underflow threshold, where e^τu = 0.The bias is input-independent and guaranteed to cross the threshold eventually, unlike query-key logits whose magnitudes vary with the input.
  • Mechanism: Different head slopes produce partial or full attention blindness: steep-slope heads underflow at shorter distances, while flat-slope heads underflow at longer distances.The simplified distances δ1 and δH mark the onset of partial and full blindness, respectively.
  • Consequences: Underflow makes every token beyond a head’s blindness distance receive zero attention weight, while the removed terms redistribute weight across the remaining tokens.Steep-slope heads therefore behave like sliding windows, a behavior previously observed without reporting the underflow mechanism.
  • Consequences: 36.6% of attention-matrix entries have crossed the underflow threshold at token distance 2048.Non-zero logits can shift the threshold locations, but any exponent scaling linearly with distance eventually underflows in sufficiently long contexts.
  • Mitigation strategies: The paper evaluates four non-exclusive mitigations: clamping, robust slopes, log-scaled distances, and soft capping.Log scaling moves δ1 from 124 to 4.37 × 10^53 under default slopes with A = 0, placing underflow beyond most contemporary model context sizes.

4 Experiments

Experiments confirm that ALiBi’s analytically predicted attention blindness appears in pretrained models and affects retrieval, while remaining difficult to separate from out-of-context degradation. Across controlled slope and mitigation studies, default ALiBi remains especially strong for needle-in-a-haystack retrieval, whereas improvements are task-dependent.

  • 4.1 Pretrained Models: Retrieval probes distinguish models more sensitively than perplexity, with needle-in-a-haystack scores substantially below passkey scores, particularly out of context.Even in-context retrieval is imperfect for the smaller BLOOM model.
  • 4.1 Pretrained Models: Underflow fractions in all three pretrained decoder models closely match the analytically expected curve, empirically confirming attention blindness beyond the 2,048-token training context.Perplexity rises slowly before spikes beginning around 1.5 times the 2,048-token training context, but shows no apparent correlation with underflow fraction.
  • 4.2 Slope Modulation: Steep slopes perform worst overall, while Safe improves passkey over ALiBi by 5 pp in context and 9 pp out of context but underperforms ALiBi on needle-in-a-haystack retrieval.Safe scores 0.48 versus 0.68 in-context and 0.02 versus 0.20 out-of-context on NIHS; Wide is strongest on QA and strong on language-generation tasks.
  • 4.2 Slope Modulation: ALiBi remains a surprisingly strong baseline on decoder benchmarks and needle-in-a-haystack retrieval, whereas the RoPE baseline fails on out-of-context retrieval.RoPE achieves the lowest validation loss and best average language-generation score, but does not preserve retrieval performance.
  • 4.3 Mitigation Strategies: Mitigation results have no clear overall winner: several combinations improve both in- and out-of-context passkey retrieval, but none surpasses ALiBi on out-of-context needle-in-a-haystack retrieval.E+S, L+S, C+E+S, and C+L+S considerably improve passkey, while ALiBi remains most effective on NIHS.
  • 4.3 Mitigation Strategies: Avoiding attention blindness alone is insufficient because flat-sloped heads function as retrieval heads, and combining mitigations often improves passkey while worsening needle-in-a-haystack performance.Clamping appears in many strong NIHS combinations, but some strategies show large seed variance in the small-scale experiments.

5 Synthesis and Discussion

ALiBi underflow is computationally wasteful and may chiefly matter during training, where exact zeros eliminate gradients that small nonzero weights would preserve. The authors recommend clamping or hard windows, view log-scaling as broadly beneficial but ineffective for NIHS, and find slope tuning promising yet incomplete.

  • Implications: Underflow affects ALiBi and related linear-bias positional encodings, wasting computation on attention pairs whose weights are discarded.Unlike an explicit sliding window, these entries are computed before being discarded.
  • Implications: Underflow may matter most during training because exact-zero weights carry no backward gradient, unlike small nonzero weights.This suggests the main practical stakes concern whether steep-sloped heads can learn information beyond their blind regions.
  • Mitigations: Clamping should be the default for a soft sliding window, while hard windows permit computation that ignores out-of-window entries and prevent unintended underflow.Both approaches avoid processing attention entries that cannot contribute to the output.
  • Mitigations: Log-scaling improves out-of-context extrapolation for passkey retrieval and many downstream tasks, but does not improve needle-in-a-haystack retrieval.Log-scaling and soft capping also appear to learn slightly more slowly, so their performance gap may shrink with further training.
  • Mitigations: Slope selection based on expected blindness distance partially succeeds, yielding the narrowest training–validation loss gap and appearing in multiple strong mitigation combinations.The approach remains promising but requires further investigation because of deterioration on needle-in-a-haystack retrieval.

6 Conclusions

The paper identifies ALiBi underflow and blind attention heads as a failure mode present in previous ALiBi models, confirming it analytically and empirically. Experiments with 148M-parameter decoders tested isolated and combined mitigation strategies, finding minor differences on downstream benchmarks and larger effects in retrieval probes.

  • 6 Conclusions: ALiBi underflow and blind attention heads occurred in all previous models using this positional encoding.The paper analyzes underflow analytically and confirms the issue empirically in pretrained models.
  • 6 Conclusions: Experiments trained numerous 148M-parameter decoder models with isolated and combined mitigation strategies.
  • 6 Conclusions: Downstream benchmarks (CS, QA, LG) differed by only 1.6 to 3.4 pp, whereas passkey and NIHS retrieval probes showed considerably larger effects.

Limitations

The study’s empirical findings are limited by experiments on 148M-parameter decoder models and by fixing architecture and training corpus. Consequently, transfer to larger models, other architectures or corpora, and broader hyperparameter configurations remains unresolved.

  • Model scale: Experiments used 148M-parameter decoder models, so not all observations may translate to larger models despite the analytically grounded underflow mechanism.The model size was chosen to balance computational costs with wide-ranging experimentation.
  • Experimental scope: Architecture and training corpus were fixed, leaving interactions between ALiBi variations, mitigation strategies, and alternative choices unexplored.The fixed setup was intended to isolate the effects of ALiBi variations and mitigation strategies.
  • Future work: A systematic, more compute-intensive hyperparameter sweep of the most promising configurations remains future work.The paper does not evaluate this broader sweep within the reported experiments.

Ethical Considerations … B.2 Softmax Invariance

The paper documents ethical and reproducibility considerations, then explains floating-point thresholds and why stable softmax prevents overflow but not underflow. It also notes that softmax invariance permits some implementations to rewrite ALiBi terms without removing this underflow behavior.

  • Ethical Considerations: The study evaluates publicly released BLOOM, Falcon-RW, and MPT models without fine-tuning or redistributing their weights.Its own experiments use FineWeb-Edu, while passkey retrieval uses template text and NIHS uses parts of FineWeb-Edu.
  • Ethical Considerations: The 148M-parameter decoder experiments used constrained model size and training budgets to keep compute feasible and costs justifiable.The authors will release code so others can adopt the setup rather than reproduce it from scratch.
  • A Floating Point Thresholds: Table 4 reports PyTorch floating-point thresholds for underflow, denormalization, and overflow on Linux 64-bit systems.The thresholds cover fp32 and bf16 numerical formats.
  • B.1 Stable Softmax: Stable softmax subtracts the maximum element from every score before computing probabilities.This formulation is described as a safer computational formulation.
  • B.1 Stable Softmax: The shift protects against overflow but does not prevent underflow.Thus, numerical loss of sufficiently small terms remains possible after stabilization.
  • B.2 Softmax Invariance: Softmax is translation invariant, satisfying softmax(x + c) = softmax(x).This invariance allows constant shifts of all logits without changing the resulting probabilities.
  • B.2 Softmax Invariance: Some BLOOM implementations use softmax invariance to rewrite the negative −m(i −j) ALiBi term as a positive slope term.Under stable softmax, the rewritten term still underflows before overflowing, so the paper retains the underflow framing.
  • B.2 Softmax Invariance: Figure 3 presents passkey-retrieval and needle-in-a-haystack curves for the pretrained models.The figure is listed in the softmax-invariance subsection.

C Implementation and Reproduction · C.1 Implementations of Existing Models · C.2 Environment

The authors implemented unsupported ALiBi mitigation mechanisms with PyTorch FlexAttention and validated it against FlashAttention. Experiments ran on a single NVIDIA H100-based Linux/CUDA environment, while existing BLOOM and Falcon implementations exhibit the same underflow behavior through stable softmax.

  • C Implementation and Reproduction: PyTorch FlexAttention implemented bias clamping, log scaling, and soft capping because FlashAttention lacked these features.The implementation was validated through preliminary correctness tests.
  • C Implementation and Reproduction: Preliminary tests directly compared the FlexAttention implementation against FlashAttention, with Flash-/FlexAttention as the only differing variable.
  • C.1 Implementations of Existing Models: BLOOM and Falcon add ALiBi bias using the shift-invariant +m_j form rather than subtracting it.Although this appears to shift failure toward overflow, stable softmax converts it back to the same underflow behavior analyzed in the main text.
  • C.1 Implementations of Existing Models: Stable softmax causes BLOOM and Falcon’s apparently overflow-oriented bias formulation to exhibit the same underflow behavior as the main analysis.
  • C.2 Environment: Experiments ran on Linux with CUDA 12.8 and Python 3.12, using a single NVIDIA H100 with 94 GiB RAM per configuration.
  • C.2 Environment: Training and evaluation, including development and failed attempts, consumed approximately 9,000 GPUh.

D Experimental Design … F Proposed Approach

The experimental design combines pretrained ALiBi model coverage, associative retrieval probes, zero-shot decoder benchmarks, and tabulated training configurations and expected blindness distances. The proposed approach is evaluated using passkey retrieval, needle-in-a-haystack retrieval, and established benchmarks.

  • D Experimental Design: The experiments cover published ALiBi models and additional models identified on Hugging Face Hub, alongside newly trained model configurations.Appendix Table 5 compiles the pretrained-model list, while Table 8 lists expected blindness distances for trained configurations.
  • D.1 Pre-Trained Models: Appendix Table 5 compiles all identified ALiBi models accompanied by a publication and all additional models found on Hugging Face Hub.The compilation is described as comprehensive to the authors’ knowledge.
  • D.2 Associative Retrieval Probes: Associative retrieval evaluation uses passkey and needle-in-a-haystack probes, with passkey and NIHS values following different repetition conventions.The PK value is stated twice, whereas the NIHS value is stated once, so this asymmetry matters when interpreting results.
  • D.2.1 Passkey Retrieval: The passkey prompt repeats the pass key before and after intervening filter text, then asks for the pass key.The filter text was adopted from Zhu et al. (2024).
  • D.2.2 Needle In Haystack: The needle-in-a-haystack prompt embeds a secret code between preceding and following content, then asks for the code.The embedded template is “[Content before.] The secret code for today is {pk}. [Content after.] The secret code for today is”.
  • D.3 Benchmarks: Standard decoder evaluation uses well-known existing benchmarks through lm-evaluation-harness in a zero-shot setting.The benchmark suite is summarized in Table 6.
  • E Hyperparameters: Training and model-architecture hyperparameters are listed in Table 7.The passage identifies Table 7 as the location of the experimental hyperparameters.
  • F Proposed Approach: Expected blindness distances for all trained model configurations are listed in Table 8.These distances characterize the trained configurations used for the proposed approach.

G Results

The results report averaged passkey-retrieval and needle-in-a-haystack curves, including all trained mitigation strategies, alongside expected failure distances and decoder benchmark outcomes.

  • G Results: Passkey-retrieval and needle-in-a-haystack curves are averaged over three runs and shown in Figures 3, 4, and 5.Figure 5 specifically presents curves for all trained mitigation strategies.
  • G Results: Table 8 lists expected failure distances for all ALiBi models trained in this work.Omitted mitigation combinations use default slopes; entries are rounded to the nearest number, with ∗ indicating δx > 10^9 and ∗∗ indicating δx ≫ 10^9.
  • G Results: The C+E+L+S configuration reports six decoder-benchmark results: 35.2±0.2, 63.6±0.7, 50.6±1.8, 65.0±1.0, 37.5±1.0, and 48.4±14.4.The passage does not identify the individual metric labels for these values.
  • G Results: The C+E+L+S configuration reports eight additional decoder-benchmark results: 49.7±1.2, 27.5±0.1, 60.4±1.3, 70.3±1.7, 29.9±1.4, 28.7±0.4, 81.3±0.6, and 50.2±0.3.The passage does not identify the individual metric labels for these values.
Loading 2608.03994v1…