PaperlayerView exact revision on arXiv

Source-linked AI summary

From Attention Masks to Inert Zero-Vector Tokens: OAttention and O-Closure for Token Dynamics

Heyang Gong

arXiv:2608.21174v1cs.AI

TL;DR

Attention masks control permitted relations but do not carry token-level non-participation. The paper introduces hidden-carrier presence and OAttention to make the origin inert while retaining standard active attention, then evaluates exactness, frozen-host compatibility, and compositional closure within explicit scope limits.

  • Problem

    Attention masks control query–source relations but do not provide a representation-carried token state that is non-participating at the attention boundary.

  • Method

    The paper assigns each hidden carrier a token-level presence coefficient and uses it to gate receiver output and source support mass in OAttention, with closure extended through compatible components.

  • Results

    The paper reports exact operator contracts, calibrated near-identity compatibility in a frozen pretrained TabPFN v3 host, and a two-block result showing whole-path closure is needed when ordinary host components can reactivate NULL.

  • Takeaways & Limitations

    Within the tested scope, representation-carried nullness is compatible with active attention and requires compositional closure beyond OAttention alone.

  • Takeaways & Limitations

    The results do not establish arbitrary-host safety, universal task-level non-regression, learned zero-attractor dynamics, or general missing-value semantics.

Abstract

from arXiv · show

Attention masks are relation-level controls: they specify which query--source pairs may interact. They do not provide a representation-carried token state that is non-participating at the attention boundary. We assign each token hidden carrier \(h_i\) an active-presence coefficient \(p_i=\lVert h_i\rVert^2/(τ+\lVert h_i\rVert^2)\). The same coefficient has two roles: it gates information emitted by token \(i\), and it determines the mass with which token \(i\) enters computations shared with other tokens. OAttention is the support-coupled attention realization of this rule. It gates the receiver output by \(p_i\) and weights source \(j\) by \(p_j\) in both the attention numerator and partition, while retaining the standard score, visibility relation, exponential competition, and value aggregation. This makes the zero-vector token a zero element and yields exact null-receiver, null-source insertion, self-attention insertion, and empty-support properties. The same token-level presence gives local O-components (OFFN, ONorm, and OInject), presence-weighted OStandardize, the O-Closure law \(M(H\oplus0)=M(H)\oplus0\), and an OTransformer by residual and compositional closure. The canonical operator is checked by contract tests and a GPU evaluation. In a zero-fine-tuning retrofit of a cloned pretrained TabPFN v3 regressor, calibrated hidden-carrier OAttention and Full-O variants change mean RMSE by $+0.088\%$ and $+0.177\%$, respectively, over 18 matched dataset--seed cases. A two-block ablation shows that OAttention alone does not preserve a NULL state through ordinary host components, whereas the OTransformer path does. These are scoped tests of exactness, active-path compatibility, and compositional necessity; they do not establish universal no-loss, arbitrary-host closure, learned attraction to the origin, or a general semantics for missing values.

1 Introduction

The paper distinguishes relation-level masks from representation-carried null states and introduces token presence as a shared basis for attention and broader operator closure. It tests exactness, frozen-host compatibility, and whole-path closure under explicitly bounded claims.

  • Motivation: Attention masks permit or forbid query–source relations but do not make non-participation a state carried by the token representation.A structural NULL instead addresses whether a representation participates at all.
  • Motivation: Standard attention treats the origin as ordinary: a zero receiver averages visible values, while a zero source changes the denominator and all pre-existing weights.These failures arise despite the zero source contributing no direct value.
  • Core construction: A single hidden-carrier presence coefficient controls both local information emission and participation in shared context support.The paper uses the same coefficient rather than introducing separate gates for each component.
  • Core construction: OAttention gates receiver output and source support mass while retaining standard scores, visibility, exponential competition, and value aggregation.Masks remain relation-level controls; token presence determines participation on otherwise visible edges.
  • Closure: O-Closure extends the presence rule through OFFN, ONorm, OInject, OStandardize, residual addition, and composition to construct a conditional OTransformer.The broader closure principle is needed because attention alone cannot ensure null consistency through ordinary host components.
  • Evaluation scope: The evaluation separates exact operator implementation, calibrated frozen-host compatibility, and whole-path closure, without claiming arbitrary-host closure or universal no-loss.The authors explicitly distinguish theorem contracts, finite-precision checks, and bounded task-metric observations.

2 Related work

Related work includes sparse, gated, routed, sink-oriented, norm-based, set-processing, and missing-data mechanisms. The paper positions its question more narrowly around an exact-zero carrier that is inert as both receiver and source while active attention retains vanilla softmax.

  • Sparse and no-op attention: Sparsemax and entmax can assign individual sources zero weight, but their simplex constraint prevents an all-zero attention row.Sigmoid attention removes row-wise normalization, while a zero dot product is not intrinsically null.
  • Sparse and no-op attention: ReLA and Softpick allow zero aggregate updates, but alter source selection or the score-normalization regime relative to vanilla softmax.The paper distinguishes these score-space no-op mechanisms from representation-carried token presence.
  • Gates and routing: Gates, routing, sinks, spare partition mass, and modified masking control branches, destinations, or relations without identifying an exact-zero carrier as a null-inserting state.These mechanisms address adjacent controls rather than the paper’s operator-level null-consistency target.
  • Norms, sets, and missing data: Norm-based mechanisms and capsule squash use vector magnitude in related ways, while permutation symmetry alone does not imply invariance to adjoining an unmasked zero element.Missing-data models further motivate separating structural NULL from informative missingness.

3 From relation masks to a null state

The paper reframes null consistency as a state-level operator contract distinct from externally supplied visibility masks. The contract requires zero tokens to be inert receivers and sources, including under insertion and empty support.

  • State versus relation: At the canonical boundary, masks are separate query–source inputs, while a zero-vector token is a hidden carrier state.Projected zeros, zero values, observed numeric zeros, and learned [MASK] tokens are distinct objects.
  • Operator contracts: Null consistency requires null-source insertion to leave every pre-existing output and old source weight unchanged.This is equivalent to removing the zero source or masking its entire source column, with old relations and metadata preserved.
  • Operator contracts: Self-attention insertion requires old outputs to remain unchanged and the inserted zero receiver to produce zero output.A full self-attention NULL has zero receiver presence regardless of visible context.
  • Operator contracts: Empty or fully masked support must return finite zero output and zero weights.The definition treats empty support as an explicit operator case rather than an ordinary softmax row.
  • Failure of standard attention: Standard attention violates null-source and null-receiver properties because softmax normalization changes with a zero source and still aggregates for a zero query.The failures are attributed to normalization topology rather than numerical instability.

4 Token presence and OAttention

Token presence is a radial, differentiable coefficient evaluated at the hidden-carrier boundary and used to couple receiver output with source support. OAttention satisfies the stated null contracts while approaching vanilla behavior for fixed nonzero states as τ tends to zero.

  • Token presence: The radial presence map is bounded in [0, 1), exactly zero only at the origin, and approaches one for fixed nonzero carriers as τ → 0+.Near the origin, attenuation is continuous rather than thresholded.
  • Token presence: Presence is evaluated before magnitude-erasing normalization, with τ as a scale hyperparameter and learned representations determining the gate.The construction adds no learned projection at this boundary.
  • OAttention: OAttention uses hidden-carrier presence for receiver output and source support mass while retaining externally supplied visibility, standard scores, and exponential competition.The source factor enters both numerator and partition, and the denominator stabilizer only totalizes empty support.
  • Null contracts: A hidden zero receiver produces zero OAttention output for all finite sources and masks.Its zero presence factor multiplies the complete aggregate.
  • Null contracts: Appending any finite number of hidden zero sources leaves every pre-existing output and old source weight unchanged.Each inserted source has zero support mass in every receiver’s numerator and partition.
  • Null contracts: Self-attention insertion preserves old outputs, gives the inserted receiver zero output, and returns finite zero outputs and weights for empty support.These properties follow by combining null-source and null-receiver behavior with the positive stabilizer.
  • Boundary choice: The canonical hidden-carrier construction differs from projection-boundary variants, where a nonzero hidden carrier may project to zero in one head.The paper keeps the canonical argument at token-level hidden-carrier presence.
  • Active-state limit: For fixed finite nonzero carriers, OAttention approaches the ordinary active-state limit as τ and the empty-support stabilizer tend to zero, although finite τ can attenuate active states.The limit does not imply exact equality at a chosen finite scale.

5 From OAttention to O-closed token dynamics

The paper extends token presence from OAttention into local and support-coupled components, then uses residual and compositional closure to define an OTransformer under explicit host-path conditions.

  • O-Closure: O-Closure requires zero absorption at the inserted position and contextual erasure on all old coordinates after compatible zero extension.The inserted coordinate remains zero, while adding a zero source leaves the old operator outputs unchanged.
  • Local and support-coupled lifts: Token-local O-components absorb zero through gated branches, while support-coupled components require presence-aware numerators and normalizers.OStandardize therefore provides insertion-invariant weighted moments and zero null output, not exact unit variance of final gated activations under the same measure.
  • Local and support-coupled lifts: OAttention preserves standard scores, visibility, exponential competition, and value aggregation while gating receivers and weighting sources by token presence.Output gating alone cannot prevent a zero source from changing a shared denominator; source presence must also enter the support measure.
  • Local and support-coupled lifts: The same presence coefficient gates locally emitted updates and supplies support mass for token-coupled computations, rather than acting as separate attention gates.Presence is recomputed at each declared hidden-carrier boundary and is shared across O-components consuming that state.
  • Residual and compositional closure: Residual addition and composition preserve O-Closure, enabling an OTransformer whose declared local paths absorb zero and support-coupled paths assign zero mass to zero states.The evaluated two-block host combines hidden-carrier OAttention with OInject, ONorm, and OFFN; OStandardize is evaluated independently.
  • Validity domain and host-model conditions: The closure result is conditional on complete module closure, compatible tokenization and metadata, and host paths such as routing, pooling, caching, and readouts satisfying the same law.The claimed exactness is evaluated at the declared hidden-carrier boundary, with floating-point equality handled using dtype-appropriate tolerances.

6 Evaluation

The evaluation separates exact operator contracts, frozen-host compatibility, and the necessity of whole-path closure. Operator checks cover finite-precision behavior, while model studies report bounded compatibility observations and a scoped OTransformer comparison.

  • Evaluation design: The evaluation tests exactness, active-path compatibility, and compositional necessity as distinct claims.The primary algebraic outcomes are null-state preservation and inert insertion; task metrics are descriptive matched comparisons.
  • Exact operator contracts: The independent GPU sweep covers multiple dtypes, attention layouts, attention modes, support conditions, gradients, and insertion positions.It evaluates fp32, bf16, and fp16 across self- and cross-attention, including empty and all-null support.
  • Pretrained active-path compatibility: +0.088% mean RMSE change was measured for hidden-carrier OAttention across 18 dataset–seed pairs in the cloned pretrained TabPFN v3 host.The retrofit leaves learned weights and preprocessing unchanged and uses no fine-tuning.
  • Pretrained active-path compatibility: +0.177% mean RMSE change was measured for Full-O across the same 18 dataset–seed pairs.Full-O adds hidden-carrier OAttention, ONorm, OFFN, and hidden-carrier gating of target-encoding additions; OStandardize is excluded from preprocessing.
  • Scope of evaluation: The reported task results are bounded inference observations rather than a universal non-inferiority test or exact empirical equality.The model-level evidence concerns minimal perturbation in the evaluated host and does not establish arbitrary-host safety.
  • Compositional necessity: OAttention alone does not repair ordinary host paths that reactivate NULL, whereas the OTransformer arm tests closure across the declared two-block path.The OA-only arm isolates this failure; all 180 OTransformer fits were finite, with maximum old-token and prediction shifts of 2.47 × 10^-3 and 2.08 × 10^-3.

7 Limitations and open questions

The paper limits its claims to declared semantics, compatible paths, and evaluated hosts. Open questions include whether the operational presence convention tracks learned meaning, preserves weak signals, and fits missing-value semantics.

  • Learned semantics: Presence is an operational convention, not evidence that training discovers or uses a meaningful null state.A zero-attractor claim would require longitudinal measurements under an explicit learning objective.
  • Finite-scale attenuation: Finite τ can attenuate weak nonzero states, and repeated gates may compound that effect.The intended regime preserves active computation when τ is below characteristic norm scales, but wider studies should examine weak-feature survival.
  • Exact-zero collisions: Cross-token centering can map an active token exactly to zero, causing the next component to treat it as NULL.A system needing semantic activity across centering should carry an explicit pre-standardization presence side channel.
  • Host scope: Whole-model closure is conditional on compatible metadata and closure of every path.Absolute-position renumbering, token merging, caches, sequence-level normalization, or a non-neutral readout can break zero extension.
  • Host scope: The current OTransformer result covers one feature-token host and excludes OStandardize, while the TabPFN retrofit covers one pretrained host without fine-tuning.These studies do not establish arbitrary-host safety or universal task-level no-loss.
  • Missingness: Observed zero, missing, NaN, learned mask states, and query placeholders have distinct semantics.Mapping missing data to an exact-zero carrier is appropriate only when missingness is declared uninformative; informative missingness needs a nonzero side channel.

8 Conclusion

The paper frames token presence as a state-level complement to relation-level attention masks and uses one coefficient to organize attention and other token dynamics. Its experiments support bounded claims about exactness, calibrated retrofit compatibility, and compositional closure.

  • Conclusion: Token presence answers whether a representation participates, complementing masks that specify which query–source interactions are permitted.The same coefficient gates receiver output and controls source mass in context support.
  • Conclusion: OAttention derives from one hidden-carrier presence coefficient while retaining standard scores, visibility, exponential competition, and value aggregation.This provides the nonlocal attention realization of the paper’s token-participation rule.
  • Conclusion: The same participation variable yields OFFN, ONorm, OInject, OStandardize, and O-Closure under residual updates and composition.These components are presented as consequences of one variable rather than independent analogous gates.
  • Conclusion: The paper’s package includes source code, tests, experiment runners, configurations, and machine-readable results for reproduction.Environment information and CPU/CUDA provenance are reported with reproduction commands.
  • Design boundary: The projection-boundary variation preserves analogous contracts at a different carrier boundary and differs from canonical hidden-carrier OAttention when projections map nonzero carriers to zero.Both forms preserve a hidden zero under zero-preserving projections.
  • Probabilistic extension: The transition-kernel analogue states that a null context leaves old-token dynamics unchanged and that the null position is absorbing.This factorization is stronger than merely assigning zero output to the new position.
  • State distinctions: A read-only query hole can retain a nonzero carrier while being excluded from source support, unlike a full structural NULL.A learned [MASK] token may likewise remain nonzero and information-bearing.
  • Closure conditions: Zero extension requires old-index metadata to remain fixed, so appending can be compatible while absolute-position reindexing may not be.OInject blocks additive metadata at an exact-zero token but does not make arbitrary re-indexing invariant.

B.2 Datasets, splits, and metrics

The learned-model evaluations use controlled dataset splits, fixed preprocessing, shared initialization, and defined metrics across adapter, zero-column, and OTransformer studies.

  • Splits and metrics: Real-data evaluations use 60% training, 20% validation, and 20% test splits, with training-fitted standardization applied unchanged afterward.Classification splits are stratified; Diabetes target transformations are fitted on training targets and inverted before reporting.
  • Splits and metrics: Metrics are accuracy, balanced accuracy, cross-entropy, RMSE, and MAE, while learned-model comparisons use seeds 11, 23, and 37 unless stated otherwise.The learned-model experiments are controlled architecture studies rather than broad benchmarks.
  • Model protocols: The hidden-carrier TabPFN comparison uses separate baseline, OAttention, and Full-O copies of one unchanged pretrained checkpoint without fine-tuning.Full-O additionally applies ONorm, OFFN, and gates to two target-encoding additions; preprocessing remains unchanged.
  • Model protocols: The OTransformer study evaluates five datasets, three seeds, three training zero counts, and four arms across 180 fits.Its host has two blocks, hidden dimension 32, four heads, and FFN dimension 128.
  • Diagnostics: The experiments track inserted-state norms, old-token and CLS shifts, prediction shifts, presence statistics, zero fractions, and finiteness.The task head lies outside the token-dynamics theorem and can produce a nonzero all-null prediction through bias.

C.1 Q/K/V-boundary reference operator properties

The Q/K/V-boundary reference study tests finite-precision null behavior, insertion invariance, active-path agreement, and finite-gate calibration on controlled hosts.

  • Reference properties: 15/15 seed–shape cases were finite, with null queries, empty support, and fully masked outputs and weights equal to zero.Null-source insertion changed old outputs by at most 2.4 × 10−7 and old weights by at most 6.0 × 10−8.
  • Reference properties: Self-attention insertion changed old outputs by at most 4.8 × 10−7 and gave the inserted query an exact zero output.Near-origin output/query scaling had a log–log slope of 1.9997 over scales 10−7 to 10−5.
  • Reference properties: The active-path output difference was at most 3.7 × 10−7 at the evaluated default scale, with tested GQA, masks, bf16, and gradients finite.These are finite-scale observations for the reference boundary design.
  • Scope: The measured near-origin slope matches the quadratic expansion, but these results do not establish the canonical hidden-carrier operator because the presence boundary differs.The study evaluates a Q/K/V-boundary design variation rather than the canonical hidden-carrier construction.
  • Calibration: The finite-τ stress matrix shows that applying the same gate to every local branch can materially alter a pretrained active path.Positive RMSE deltas and negative R2 deltas indicate degradation relative to the unmodified checkpoint.

C.5 Training-time exact-zero columns

The training-time zero-column study reports a matched canonical hidden-carrier comparison and places its interpretation within controlled, non-universal evaluation limits.

  • Training-time comparison: Table 10 reports mean test metrics over three seeds for standard versus hidden-carrier OAttention in the canonical training-time zero-column study.For Diabetes, lower RMSE is better.
  • Scope: The study was not designed with a formal non-inferiority margin, so task- and seed-level differences cannot support a universal no-loss statement.The comparison is bounded and matched rather than a universal non-regression test.
  • Scope: Component checks verify implementation agreement with displayed definitions but do not show that every possible wrapped module is safe.They test biased FFNs, affine normalizers, arbitrary additive encodings, active-path convergence, gradients, dtypes, shapes, insertion, and ablations.

D Reproducibility artifacts

The paper supplies machine-readable experiment artifacts, scripts, receipts, and source digests supporting its numerical results and reproducibility workflow.

  • Code and artifacts: The zero_activity_attention package contains the implementation and principal entry points for operator, adapter, FT-transformer, zero-column, OStandardize, and OTransformer experiments.The listed scripts correspond to the paper’s major evaluation suites.
  • Code and artifacts: Publication numerical values are supported by machine-readable artifacts stored under experiments/results/.The artifact list includes operator sweeps, adapter matrices, FT and zero-column suites, OStandardize, OTransformer, and TabPFN results.
  • Code and artifacts: The artifact inventory includes separate receipts for hidden-carrier OAttention, systematic operator sweeps, adapter non-regression, FT-transformer, zero-column, OStandardize, and OTransformer evaluations.The paths identify the stored JSON outputs for each experiment family.
  • Integrity metadata: Source and runner integrity are documented through SHA-256 prefixes for canonical, receipt, historical operator, OStandardize, and OTransformer artifacts.The complete digests are stored in the corresponding receipts.
Loading 2608.21174v1…