PaperlayerView exact revision on arXiv

Source-linked AI summary

Avey-B

Devang Acharya, Mohammad Hammoud

arXiv:2602.15814v1cs.CLcs.AI

TL;DR

Long-context bidirectional encoders need an alternative to self-attention’s quadratic costs. This paper reformulates Avey as Avey-B with architectural changes for encoder-only contextualization, and reports stronger benchmark performance with better long-context efficiency than Transformer baselines.

  • Problem

    Full self-attention enables bidirectional contextualization but incurs quadratic time and memory costs, while efficient alternatives have been little adapted to bidirectional encoder-only settings.

  • Method

    Avey-B adapts Avey into a bidirectional encoder using decoupled static and dynamic parameterizations, row-normalized similarity, and neural compression of retrieved context.

  • Results

    Avey-B consistently outperforms the evaluated Transformer encoders on token-classification and information-retrieval benchmarks and is faster across 128–96 K tokens.

  • Takeaways & Limitations

    The results support retrieval-conditioned, non-attention architectures as a viable route to strong bidirectional encoders.

  • Takeaways & Limitations

    Efficiency comparisons use Avey-B with torch.compile or eager PyTorch, whereas Transformer baselines use mature FlashAttention-optimized implementations; long-context extrapolation may affect downstream effectiveness.

Abstract

from arXiv · show

Compact pretrained bidirectional encoders remain the backbone of industrial NLP under tight compute and memory budgets. Their effectiveness stems from self-attention's ability to deliver high-quality bidirectional contextualization with sequence-level parallelism, as popularized by BERT-style architectures. Recently, Avey was introduced as an autoregressive, attention-free alternative that naturally admits an encoder-only adaptation. In this paper, we reformulate Avey for the encoder-only paradigm and propose several innovations to its architecture, including decoupled static and dynamic parameterizations, stability-oriented normalization, and neural compression. Results show that this reformulated architecture compares favorably to four widely used Transformer-based encoders, consistently outperforming them on standard token-classification and information-retrieval benchmarks while scaling more efficiently to long contexts.

1 INTRODUCTION

Avey-B reformulates the attention-free Avey architecture as a bidirectional encoder, addressing the limited adaptation of efficient long-context alternatives to encoder-only settings. Its decoupled parameterization, normalization, and neural compression improve effectiveness and efficiency against Transformer encoders.

  • Bidirectional Transformer encoders provide strong contextualization but full self-attention has quadratic time and memory costs, limiting practical long-context deployment.
  • Avey-B adapts Avey to encoder-only bidirectional contextualization using decoupled static and dynamic parameterizations, row-normalized similarity scores, and neural compression.
  • Decoupling prevents fixed weights from reversing relevance-based contributions, while row normalization bounds per-row gain and improves numerical conditioning and trainability.
  • A neural compression module reduces each split and its top-k retrieved splits to one split-sized representation, making neural-processor compute independent of k.
  • Avey-B outperforms BERT and NeoBERT across all evaluated benchmarks and consistently surpasses RoBERTa and ModernBERT on token-classification and information-retrieval tasks.It does so despite pretraining on approximately 11× fewer tokens than ModernBERT.
  • At N = 96 K, Avey-B is 3.38× faster than ModernBERT and 11.63× faster than NeoBERT, with its efficiency advantage widening as sequence length increases.

3 BACKGROUND

Avey retrieves relevant sequence splits with a ranker and processes them through an enricher, contextualizer, and fuser. The neural processor expands, selectively contextualizes, and recombines token features while maintaining favorable scaling properties.

  • Avey partitions an input of length N into S-token splits and selects the top-k preceding splits for each target split using MaxSim relevance scores.
  • The ranker normalizes selected-split scores, weights the retrieved splits, and concatenates them with the current split to emphasize relevant context.
  • The ranker runs once per full forward/backward pass, independent of neural-processor depth, with training-time matching cost O(N^2d).
  • Each neural-processor layer contains an enricher, contextualizer, and fuser operating on the current split and weighted retrieved context.
  • The enricher expands each token embedding from d to m dimensions, bypassing a head stream while sending a tail stream to contextualization.The bypass preserves token-specific features and mitigates over-smoothing as layers increase.
  • The contextualizer uses cosine-similarity-based dynamic interactions and a learned cross-embedding matrix, while the fuser projects combined features back to dimension d through a residual addition.
  • Aggregated training complexity is O(N^2d), while autoregressive inference complexity is O(N) because only the most recent split is contextualized.

4 AVEY-B

Avey-B adapts Avey into a bidirectional encoder by removing causal masking and separating static transformations from dynamic similarity-based mixing. It also compresses retrieved split context before neural processing to reduce redundant computation.

  • Avey-B removes the autoregressive mask, allowing token representations to condition on both left and right contexts while retaining ranker-selected global access.
  • 4.2 DECOUPLED PARAMETRIZATION: Decoupled static and dynamic layers prevent learned weights from inverting similarity-based contribution rankings while still allowing representation shaping.
  • 4.2 DECOUPLED PARAMETRIZATION: Row-wise normalization produces a row-stochastic similarity operator that bounds per-row gain and improves numerical conditioning and trainability.
  • 4.3 NEURAL COMPRESSION: A neural compressor maps each retrieved (k + 1)S-token block back to S representative tokens before neural processing.
  • 4.3 NEURAL COMPRESSION: Reducing contextualized tokens per split from (k+1)S to S improves throughput by 4.37× while leaving asymptotic complexity quadratic in sequence length N.

5 EXPERIMENTS

The experiments evaluate Avey-B against established and recent Transformer encoders across four downstream categories and controlled efficiency settings. Avey-B achieves its strongest comparative results on token classification and information retrieval, while scaling more favorably to long contexts.

  • 5 EXPERIMENTS: The evaluation covers sequence classification, token classification, question answering, and information retrieval using three benchmarks per category.
  • 5 EXPERIMENTS: Models are compared using median scores across 10 seeds at the best of four learning rates, with accuracy, F1, and NDCG@10 as task-specific metrics.
  • 5.3 EFFECTIVENESS: At base scale, Avey-B surpasses BERT and NeoBERT across all task categories and delivers the strongest results on token classification and information retrieval.
  • 5.3 EFFECTIVENESS: At large scale, Avey-B again surpasses BERT and NeoBERT across all categories and exceeds every Transformer baseline on token classification and information retrieval.
  • 5.4 EFFICIENCY: Under eager execution, Avey-B has decay exponent αAvey-B = 0.33 and maintains stable throughput across the tested sequence-length range, while both Transformer baselines encounter out-of-memory failures.

6 CONCLUSION

The paper presents Avey-B as an attention-free bidirectional encoder with architectural refinements for parameterization, normalization, and compression. It reports consistent gains on token classification and information retrieval and broader gains over BERT and NeoBERT.

  • 6 CONCLUSION: Avey-B contributes decoupled static and dynamic parameterizations, row-normalized similarity, and a neural compression module.
  • 6 CONCLUSION: Avey-B consistently outperforms Transformer-based encoders on token classification and information retrieval while outperforming BERT and NeoBERT on every evaluated benchmark.
  • 6 CONCLUSION: The findings motivate further study of retrieval-conditioned, non-attention architectures for bidirectional encoding.

7 REPRODUCIBILITY

The paper states that its results are fully reproducible and provides detailed architectural and experimental specifications together with publicly available source code.

  • 7 REPRODUCIBILITY: The repository publicly provides scripts, exact experiment configurations, and materials for pretraining, fine-tuning, and evaluation.

A MONOTONICITY UNDER DECOUPLING

Avey-B’s decoupled dynamic layers provide a relevance-monotonic update under explicit nonnegativity, positive-normalization, and monotone-activation assumptions. Subsequent static layers preserve this guarantee because they do not alter the preceding relevance scores.

  • Dynamic-layer assumptions: Avey-B’s dynamic layer is analyzed under nonnegative similarities, positive normalization, and a monotone nondecreasing activation.The enricher uses ReLU2 for nonnegative similarities, normalization includes a positive stabilizer, and the activation is monotone.
  • Dynamic-layer monotonicity: For a fixed target, increasing one token’s similarity weakly increases its normalized weight and weakly decreases or leaves unchanged the others’ weights.This establishes self-monotonicity and cross-token monotonicity in the normalized similarity scores.
  • Dynamic-layer monotonicity: Nonnegative coefficients and a monotone activation ensure that the dynamic update remains monotone with respect to relevance.A more relevant token cannot reduce or flip the sign of its contribution within the dynamic layer.
  • Static-layer preservation: A following static layer cannot violate the preceding dynamic layer’s monotonicity because its learned parameters are independent of the relevance scores.The static layer reshapes representations for later layers but does not retroactively modify earlier scores or inequalities.

B PRETRAINING METHODOLOGY

Avey-B is pretrained with a BPE tokenizer and masked-language-modeling setup, then evaluated against Transformer encoders across four downstream task categories. The ablations identify effective choices for ranker direction, layer arrangement, normalization, context coverage, and masking.

  • Pretraining setup: Avey-B uses a 50,368-token BPE vocabulary derived from OpenAI’s p50k base tokenizer, retaining BERT-style special tokens.Only the [MASK] token is used during pretraining.
  • Pretraining setup: The models are pretrained at 165M and 391M parameters for 180B FineWeb tokens on 16 NVIDIA H200 GPUs.Both sizes use a global batch size of 512K tokens and AdamW optimization.
  • Ranker direction: A bidirectional ranker underperforms a unidirectional ranker across all task categories, with the largest drops on QA and IR.QA declines from 51.07 to 36.51 (∆= −14.56), while IR declines from 74.82 to 66.20 (∆= −8.62).
  • Layer arrangements: The interleaved S → D arrangement performs best on average for SC, TC, and QA, while all-dynamic layers perform worst across categories.A static front layer may provide a stable representational scaffold before input-dependent mixing.
  • Normalization: Divide-by-sum normalization performs strongest overall, preserving similarity ordering and margins while retaining sign information that softmax loses.Softmax performs better on IR, whereas RMS normalization underperforms divide-by-sum across categories.
  • Context and masking: Performance generally peaks when effective context C = S (k+1) matches or closely approximates training length N, with N=2048 best for SC, QA, and IR.TC peaks at N=512 but remains within +0.09 points of N=1024.
  • Context and masking: Masking around 20%–30% is near-optimal for both model sizes, although IR benefits most from 50% masking and aggressive masking can destabilize training.The paper therefore adopts 20% masking for both models.

H ABLATION STUDY

The ablations support decoupling, row-wise normalization, residual connections, and shallow retrieval as core Avey-B design choices. Neural compression delivers a large efficiency gain with modest task-dependent accuracy changes, while deeper retrieval is both slower and less effective.

  • Ablation design: Avey-B ablations cover decoupling, row-wise normalization, neural compression, residual connections, and removing the ranker.The study fixes sequence, split, retrieval, interleaving, and normalization settings to selected configurations before evaluating these components.
  • Decoupling and normalization: Removing decoupling reduces SC, TC, QA, and IR performance by 1.43%, 2.12%, 2.53%, and 7.40%, respectively.The result supports separating similarity scoring from neural learning.
  • Decoupling and normalization: Removing row-wise normalization causes larger degradations of 3.55%, 0.87%, 7.65%, and 15.33% across SC, TC, QA, and IR.The largest reported decline occurs on IR.
  • Neural compression: Neural compression reduces contextualized tokens per split from (k+1)S to S and improves throughput by 4.37×.Compression has negligible or positive average effects on SC and TC but causes modest drops on QA and IR.
  • Residual connection: Removing the residual connection degrades every benchmark by an average of 3.38%, indicating its role in preserving local signal.The residual connects compressor output with the current split’s original tokens.
  • Ranker: Removing the ranker causes universal degradation with a 7.46% average drop, confirming retrieval’s importance for Avey-B effectiveness.The ranker runs once before the neural processor and retrieves using shallow initial embeddings.
  • Ranker: Retrieval at every layer produces a 27.28% average performance drop and slows efficiency by 5.9×.These results support shallow, one-time retrieval over deeper-layer retrieval.
  • Caveat: Top-k retrieval can include weakly relevant or noisy tokens because the ranker lacks a hard relevance threshold.The nearest boundary is that larger retrieved sets are not guaranteed to contain only relevant content.

I LATENCY RESULTS

Avey-B has lower latency and better long-context scaling than the Transformer encoders, including without a fused kernel. Its latency advantage grows as sequence length increases.

  • Avey-B achieves consistently lower latency than ModernBERT and NeoBERT across evaluated sequence lengths.The comparison includes both eager and torch.compile Avey-B implementations, alongside optimized and unoptimized Transformer variants.
  • 0.58: Avey-B-eager has the shallowest latency-growth exponent across all configurations.
  • 1.42 and 1.63: ModernBERT-sys-unoptimized and NeoBERT-sys-unoptimized show substantially steeper latency growth and fail out of memory before the maximum sequence length.

J CROSS-SEED VARIANCE ANALYSIS

Across random-seed evaluations, Avey-B is among the most statistically consistent encoders. The analysis attributes this robustness to architectural choices that separate parameter roles, stabilize normalization, and remove irrelevant computations.

  • Avey-B ranks second in overall variance at base scale, behind RoBERTa and ahead of ModernBERT, BERT, and NeoBERT.
  • Avey-B maintains tightly concentrated variances across most benchmarks, except UNER, where variability reaches SD = 2.65.
  • At large scale, ModernBERT shows substantial instability on ReCoRD, UNER, and NQ, while BERT reaches SD = 8.24 on QQP.
  • The paper attributes Avey-B’s robustness to decoupled static and dynamic layers, row-normalized similarity matrices, and neural compression.
  • −1.44, −0.77, −1.79, and −0.51 points: unsigned static weights reduce effectiveness on SC, TC, QA, and IR, respectively.
  • Signed static weights preserve inhibitory effects, which support representational contrast and disambiguation, especially in QA.

L COUPLED VS. DECOUPLED LAYERS: A STATISTICAL ANALYSIS

Coupling static projections with similarity scores biases them toward homogeneous, nearly nonnegative transformations. Decoupling preserves balanced signed weights, larger variation, locality-like structure, and relevance monotonicity.

  • Coupled projections develop a strong positivity bias, with the fraction of positive entries reaching one in layers 8 and 13.
  • Residual negative weights in coupled layers can invert highly positive similarities into negative contributions, violating relevance monotonicity.
  • Decoupled dynamic layers enforce nonnegative, normalized similarity weights, while separately learned static layers retain balanced positive and negative weights.
  • Decoupled matrices exhibit larger weight fluctuations than coupled matrices, admitting stronger positive and negative values and potentially greater representational flexibility.
  • Both variants learn Toeplitz-like, diagonally dominant, near-diagonal static matrices indicative of locality.
  • Overall, decoupling promotes diverse, sharper projections while preserving monotonicity with respect to relevance.

M LONG-RANGE BENCHMARK RESULTS

Avey-B maintains strong long-context performance far beyond its 2,048-token training window, whereas Transformer baselines are limited by declining accuracy, fixed context windows, or memory failures. It sustains high accuracy up to 96k tokens on the reasoning-intensive benchmark.

  • The benchmark combines semantic retrieval with position-sensitive reasoning, using 40% single-needle and 60% two-needle examples.
  • Avey-B generalizes beyond its trained 2,048-token context window without a fixed maximum sequence length, while baselines are evaluated only within shorter trained windows.
  • On NIAH-1, Avey-B base and large lose only 3–4 points across a 96× sequence-length increase from 1k to 96k tokens.
  • 10–12 points: ModernBERT trails Avey-B at short contexts, while ModernBERT large fails at 8k tokens because of out-of-memory issues.
  • On NIAH-2, Avey-B base declines from 78.3 at 1k to 71.9 at 96k, while Avey-B large declines from 78.9 to 74.5.
  • NeoBERT remains competitive at 1–4k tokens but cannot operate beyond its 4k-token window, preventing long-context evaluation.
  • Avey-B is the only evaluated model sustaining high accuracy up to 96k tokens on the reasoning-intensive NIAH question-answering benchmark.
Loading 2602.15814v1…