PaperlayerView exact revision on arXiv

Source-linked AI summary

Transformers as Generic Deep Neural Networks: Exact Compilation, the Cost of Removing Activation-Activation Products, and Multi-Agent Attention Clustering

Sergey Salishev, Zeev Volkovich, Oleg Granichin

arXiv:2610.28958v1cs.LGcs.MA

TL;DR

Transformers are deployed with dedicated attention and normalization hardware operators, while their post-training quantization relies on Transformer-specific algorithms. This paper compiles Transformer blocks exactly into generic DNNs and shows that activation products remove a lower-bound cost of product-free compilation.

  • Problem

    Transformers are treated as a special model class, with dedicated NPU operators for attention and normalization and Transformer-specific post-training quantization algorithms.

    Source: Introduction · p. 1

  • Method

    The paper constructs exact generic-DNN compilations of Transformer blocks and analyzes product-free costs, activation-product extensions, and top-k attention with estimated tail mass.

    Source: Introduction · p. 2

  • Results

    Product-free transform-wise compilation requires at least N^2 scalar nonlinear evaluations per head for dense scores, while adding activation products removes that cost.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    The elementwise Hadamard product suffices to remove the added pointwise-work cost and is present on every surveyed NPU.

    Source: Introduction · p. 2

Abstract

from arXiv · show

Transformers are deployed as a special model class. Neural processing units (NPU) add dedicated operators for attention and normalization, and quantization relies on Transformer-specific algorithms. We show the computation does not force this. Every Transformer block, with attention, LayerNorm and a GELU MLP, compiles exactly into a generic deep neural network built from fixed affine maps, routing and the square, sigmoid and inverse square root, plus GELU or its sigmoid approximation. Without activation-activation products, every transform-wise compilation adds at least $N^2$ scalar nonlinear evaluations per head for dense scores. Keeping that one operation type removes this cost; its elementwise form, present on every surveyed NPU, suffices with the sigmoid and inverse square root. For top-$k$ attention with exchangeable tail scores, we prove that exact or conditional-mean tail mass gives an unbiased, minimum-mean-square-error estimate of the attention distribution, while tail drop is conditionally biased. When queries or agents cluster, centroids choose the keys first, reducing the number of computed scores from $N^2$ to $Nk$ per head. On DeiT-Tiny, MambaIRv2-Light and Pythia-70M the compiled models match their references with all keys, standard post-training quantization applies, and the exact tail-mass estimate beats tail dropping with half as many keys on DeiT-Tiny and, for $k\le8$, on Pythia-70M.

Introduction

The paper argues that Transformer blocks can be compiled exactly as generic DNNs and studies the cost of omitting activation products alongside methods for reducing attention computation. It also examines reduced-key attention with tail-mass estimates and clustered-query routing, while reporting that compiled models match their references with all keys and support standard post-training quantization.

Introduction

  • Deployment often splits unsupported operations across accelerator and host, while Transformer PTQ uses dedicated algorithms [2–14].Examples include Edge TPU and Jetson DLA limitations and phone implementations that leave LayerNorm and attention on CPU or GPU [4].Source: Introduction · pp. 1–2

Top-k attention with a tail-mass estimate

  • Under exchangeable tail scores, exact or conditional-mean tail mass gives an unbiased, minimum-mean-square-error attention estimate, whereas tail dropping is conditionally biased.Exact mode still computes all N^2 scores for the dense normalizer.Source: Introduction · p. 2, Top-k attention with a tail-mass… · p. 8

A generic DNN vocabulary

  • The strict DNN vocabulary combines fixed affine maps and routing with square, sigmoid, inverse-square-root, and GELU operations.The sigmoid supplies exponentials, while squaring the inverse square root supplies reciprocals for positive inputs.Source: A generic DNN vocabulary · p. 4

Every Transformer block compiles exactly

Strict compilation pays an unavoidable pointwise cost

  • For free inputs, every product-free exact compilation needs at least N^2 unary evaluations for scores and Nd for value mixing per head, regardless of depth or width.These bounds count evaluations rather than nodes, so fusing nodes does not lower them.Source: Strict compilation pays an unavoidable… · p. 5

Activation–activation products remove the cost

  • Activation products enable zero-added-work compilation with matrix multiplication; Hadamard products alone suffice with sigmoid and inverse square root, but softmax adds 2N^2 unary evaluations per head.Theorems 2 and 3 make activation–activation products necessary and sufficient for transform-wise compilation without added pointwise work.Source: Activation–activation products remove… · p. 6

Attention weights cluster on a few keys

Top-k attention with a tail-mass estimate

  • With exchangeable tail scores, the conditional-mean tail-mass estimate is unbiased and minimum-MSE for the attention distribution and output; tail drop is conditionally biased unless expected tail mass is zero.The theorem concerns one head’s distribution and output, not whole-model loss; exact mode also has this optimality conditional on the observed tail mass.Source: Top-k attention with a tail-mass… · p. 8

Centroids factor the selection scores

The compiled models match their references

  • With all keys kept, compiled models match their references: 91.36% vs 91.36% on Pet, 72.01% vs 72.02% on ImageNet, 32.51 dB vs 32.51 dB on Set5, and 4.5496 vs 4.5496 token NLL on WikiText-2.The strict and extended graphs compute the same model exactly; only model changes such as top-k attention and rounding change accuracy.Source: The compiled models match their… · p. 10

The strict graph pays the predicted cost

  • The strict graph runs at 1.2–2.4× reference latency and uses 2.3× the extended graph’s peak memory at top-64; its measured slowdown comes from construction and memory writes, not the theoretical bound.A per-module profile attributes 73–77% of added time to Score and Mix.Source: The strict graph pays the predicted cost · p. 10

The elementwise product removes most of the strict cost

  • With top-k support and k ≤64, the Hadamard graph runs at 1.2–1.3× extended-graph latency versus 1.4–1.5× for strict, and uses 2.2 × 10^6 unary evaluations per image at k = 16 versus 1.1 × 10^8 for strict.All ten surveyed NPUs multiply activations elementwise, so none is product-free; the Hadamard graph runs wherever the strict graph does.Source: The elementwise product removes most… · p. 10, NPUs provide the Hadamard basis · p. 11

Standard quantization applies to the compiled graph

  • Standard post-training quantization changes strict-graph accuracy by −1.47 to −0.05 points with per-channel scales and −1.06 to −0.14 with per-tensor scales, versus −2.5 to −3.9 points on the extended graph.The method uses convolutional-network PTQ without training, reconstruction, or a Transformer-specific quantizer.Source: Standard quantization applies to the… · p. 11

The tail-mass estimate beats tail drop

  • On DeiT-Tiny/Pet, exact mode at k beats tail drop at 2k for every tested k ≤32; on Pythia-70M it does so for every k ≤8, and on MambaIRv2-Light it is best for every k < N.On DeiT-Tiny/Pet, calibrated mode also beats tail drop at 2k for k ≤16; on MambaIRv2-Light it falls below tail drop for k ≥16 because one constant per block fits the per-query tail mass poorly.Source: The tail-mass estimate beats tail drop · p. 12

Clustered queries cut the cost of the strict graph

  • At k = 16, routed strict DeiT-Tiny uses 2.4 × 10^7 unary evaluations and 2.2× reference memory, versus 1.1 × 10^8 and 8.6× for exact top-k; routing retains 83–95% of exact top-k attention mass at k = 8–64.At k = 32, routed support with twice the keys reaches 85.96% accuracy using 37% of exact top-16 unary evaluations and 41% of its memory; Pythia-70M queries cluster more tightly, but attend to different keys within a cluster, and tail-drop perplexity rises 9% at k = 32.Source: Clustered queries cut the cost of the… · pp. 12–13

Discussion

The discussion connects activation products to efficient NPU execution and extends the construction to other blocks and multi-agent attention. It also bounds the theorem’s scope and identifies on-device NPU evaluation as future work.

Discussion

  • A Hadamard product removes the pointwise cost priced by Theorem 2; a MAC array accepting two activation operands keeps QK^T and PV products inside the array.The discussion says an NPU needs no dedicated attention, softmax, or normalization unit to run Transformers.Source: Discussion · p. 13
  • Theorem 2 covers transform-wise compilations that retain attention contractions as subgraphs, not compilations that never form QK^T.Corollary 1 also requires a MAC array that outpaces the pointwise unit by more than 2Nd/(N + d).Source: Discussion · p. 13
  • RMSNorm compiles like LayerNorm without the mean, while SwiGLU uses polarization in the strict basis and a Hadamard node in the other two bases.The construction also handles the SiLU activation in SwiGLU.Source: Discussion · p. 13
  • For multi-agent systems, top-k attention corresponds to fixed-neighbor flocking, while the tail-mass term describes far agents by total weight and mean velocity.The paper identifies a swarm optimizer using top-k attention and a tail-mass term as a multi-agent counterpart of its results.Source: Discussion · pp. 13–14
  • The reported latency and memory are GPU measurements, and NPU placement is based on documented operator support; compiling and running the graphs on NPU devices remains future work.Source: Discussion · p. 13
  • At k = 0, all attention weight lies on the uniform tail, which tail drop cannot express; Table 5 gives Set5 references of 32.51 dB at ×4 and 38.24 dB at ×2.Source: Table 5 · p. 14

Methods

The Methods section presents exact Transformer compilation into strict and extended graph bases, analyzes product-free compilation costs, and develops tail-mass and centroid-routing methods for top-k attention. It also specifies the fine-tuning, quantization, timing, and numerical procedures used to assess compiled models and count unary evaluations across alternative graphs.

Proofs of Theorem 1, Propositions 1 and 2 and Theorem 3

  • Exact composition of transform realizations compiles the block in both bases; the extended construction preserves source pointwise work and multiply–accumulate counts.The strict graph replaces products with polarization graphs, while the extended graph uses Hadamard products for Score and Mix and MatMul or BMM nodes.Source: Proofs of Theorem 1, Propositions 1… · pp. 16–17

Proof of Theorem 2

  • Product-free exact compilation requires at least N^2 pointwise nonlinear evaluations for dense scores and Nd for value mixing; across h heads, the bound is h(N^2 + Nd).The comparison with an extended graph’s 2hN^2d multiply–accumulate steps depends on κ > 2Nd/(N + d).Source: Proof of Theorem 2 · p. 18

Proofs for clustered attention

  • For exchangeable tail scores, conditional-mean or exact tail mass yields an unbiased minimum-mean-square-error attention estimate, while tail drop is conditionally biased.Conditional expectation of the attention weights gives the estimate; with q = 0, the conditional output bias is E[τ | O](a − ū).Source: Proofs for clustered attention · p. 19

Models and data

  • The evaluated models are DeiT-Tiny on Pet and ImageNet-1k, MambaIRv2-Light on Set5, and Pythia-70M on WikiText-2; DeiT-Tiny’s Pet head is fine-tuned for 20 epochs.The compiled DeiT-Tiny Pet model is also distilled against the frozen reference for one epoch using AdamW.Source: Models and data · p. 20, Distillation and fine-tuning · p. 21

Tail-mass calibration

  • Calibrated mode sets each block’s tail mass q to the mean dense-softmax mass outside the top-k support on calibration minibatches; exact mode computes τ per query at runtime.Calibration uses Pet training, ImageNet training, and DIV2K minibatches; Pythia-70M has no calibrated mode.Source: Tail-mass calibration · p. 20

Query-centroid routing

  • Centroid-based routing selects each query’s top-k keys before scoring, with selected-score and mixing costs proportional to Nkd rather than dense query-key scoring.The method fits 32 centroids; at k = 16, routed support mass is 76% and 92% of top-k mass on DeiT-Tiny with 8 and 128 centroids, respectively, and 89% and 93% on Pythia-70M.Source: Proofs for clustered attention · p. 19, Query-centroid routing · p. 20
  • With routed support, strict-graph GELU latency is 4.19, 3.89, and 5.41 times reference at k = 16, 32, and 64, versus 4.34, 4.58, and 6.19 for exact top-k.The extended graph gives 3.35 to 4.37 times reference with routed support versus 3.28 to 4.88 with exact top-k; peak memory changes by less than 5%.Source: Query-centroid routing · p. 20

Distillation and fine-tuning

  • MambaIRv2-Light is fine-tuned for three AdamW epochs on L1 loss, while Pythia-70M is evaluated without training.The MambaIRv2-Light checkpoint is selected by training PSNR on one center crop per DIV2K training image; trainable tail mass q is clamped to [0, 1] in the forward pass.Source: Distillation and fine-tuning · p. 21

Post-training quantization

  • The PTQ graph quantizes linear, convolution, product, and coefficient-tensor affine nodes, while unary, routing, and plain sums or means have no separate quantizer.Calibration uses two forward passes over the full Pet training split; activations use symmetric per-tensor scales, and linear or convolution weights use symmetric per-output-channel scales. The resulting graph passes integer codes between quantized units, with NPU units implemented as integer multiply–accumulate and requantization [53] and unary approximations [48, 49].Source: Post-training quantization · pp. 21–22
  • For signed 8-bit operands, square identities recover integer products exactly when intermediate steps neither overflow nor requantize.A shared square table needs 512 signed-index or 257 magnitude-index entries; the stated 32-bit accumulator supports dot products of length d < 2^17.Source: Post-training quantization · p. 22

Timing and unary counts

  • With all keys, DeiT-Tiny uses 6.02 × 10^6 unary evaluations per image for the Hadamard graph, 1.96 × 10^8 for strict GELU, and 4.17 × 10^6 for the extended graph.The extended count includes unary square nodes for 25 LayerNorms; the strict sigmoid-approximation graph adds 8Nhd more per block.Source: Timing and unary counts · p. 22

Numerical conventions

  • In float64, three all-key DeiT-Tiny graphs reproduce reference logits on 16 Pet test images to 6 × 10−8.The comparison uses reference weights rounded to bf16; the reported Table 2 differences therefore arise from bf16 weights and arithmetic.Source: Numerical conventions · p. 23

Data availability

The paper uses public datasets across image classification, super-resolution, and language modeling, with pretrained weights drawn from established model repositories.

Data availability

S1. Tightness and scope of the bound

The product-free lower bound is tight in order for fixed d, but its strength depends on the allowed nonlinearities and on treating attention contractions as transforms. Square-only compilation connects a stronger bound to an open matrix-multiplication question, while unrestricted nonlinearities admit an encoding-based construction.

S1. Tightness and scope of the bound

  • For fixed d, the N^2d lower bound is tight in order for square-only graphs, but for d=N a bound of order N^(2+δ) would imply a matrix-multiplication exponent of at least 2+δ.That stronger exponent remains an open problem; fast matrix multiplication yields strict graphs with PW = O(n^2.373) for square-only nonlinearities.Source: S1. Tightness and scope of the bound · p. 29
  • On a bounded box, unrestricted scalar nonlinearities permit a product-free score graph with N^2+2Nd unary evaluations, limiting improvements over Theorem 2 to the additive 2Nd.The construction affinely maps inputs into [0, 1), interleaves their binary digits, then decodes each score with one unary map.Source: S1. Tightness and scope of the bound · p. 29
  • The encoding construction reaches its bound by first encoding inputs; one unary evaluation of a linear form cannot compute a product on an open set.Such a composition is constant on hyperplanes.Source: S1. Tightness and scope of the bound · p. 29
  • Theorem 2 does not cover compiling a whole block as one map: its nonlinear rank is at most the number of outputs, 3.78 × 10^4 for DeiT-Tiny.The theorem instead concerns attention contractions as transforms, the form used by Transformer implementations and NPU operator sets.Source: S1. Tightness and scope of the bound · p. 29

S2. NPU operator support

The surveyed accelerator documentation supports the strict graph on more devices than the extended graph, while every surveyed accelerator supports elementwise activation products. These are documented operator-support findings; compiling and running the graphs on the devices remains future work.

S2. NPU operator support

  • The table reports documented operator support for dense compiled graphs, defining “Whole” as support for every graph operation on the accelerator; device compilation and execution are future work.Open-source placement comes from compiler or runtime code, while closed toolchains are assessed from vendor documentation.Source: S2. NPU operator support · p. 29, Table S1 · p. 30
  • The strict graph runs whole on seven listed accelerators, while the extended graph runs whole on Ethos-U, VeriSilicon, TDA4VM, and RK3588, plus R-Car with its DSP.The strict graph requires squares, sigmoid, and inverse square root; R-Car lacks inverse square root, and some NVIDIA toolchains leave that operation and LayerNorm work to the host.Source: S2. NPU operator support · p. 30
  • Every surveyed accelerator supports elementwise activation multiplication, so each provides the Hadamard basis; with sigmoid and inverse square root, it also runs the rest of the block.The Hadamard basis computes QK^T and PV without pointwise work using a broadcast product followed by a fixed sum.Source: S2. NPU operator support · p. 31
  • TopK runs on the TDA4VM accelerator and R-Car DSP; elsewhere selection stays on the host, although Ethos-U and VeriSilicon can use masked k arg-max passes.The Edge TPU compiler splits only once, so the extended graph moves to CPU from its first unsupported attention score.Source: S2. NPU operator support · pp. 30–31

S3. Why the bound counts pointwise work

The communication bound concerns words moved between fast and slow memory, not the unary-evaluation count. Fusion and tiling give the strict score computation the traffic order of native matrix multiplication without changing its pointwise work.

S3. Why the bound counts pointwise work

  • IOL(G) is the minimum number of words moved between fast memory of L words and slow memory over schedules of graph G.With bandwidth β, schedule time is related to this communication measure.Source: S3. Why the bound counts pointwise work · p. 31
  • The unfused strict graph writes N^2d intermediate values q_if + k_jf, giving arithmetic intensity Θ(1) and traffic far above IOL.These writes are schedule-dependent rather than a property of the basis.Source: S3. Why the bound counts pointwise work · p. 31
  • Tiling and fusing the strict score expansion gives the same traffic order as native QK^T, while fusion changes neither unary evaluations nor the unit that performs them.The same tiling applies to a Hadamard product followed by a fixed reduction, giving it the matrix-product access pattern.Source: S3. Why the bound counts pointwise work · p. 31

S4. Where the compiled graph spends its time

At k=16, compiled forward-pass timings place the reference ahead of the extended, Hadamard, and strict graphs. Eager module profiling shows substantial strict-graph score costs, while routing reduces computed-score time and memory at the cost of centroid selection.

S4. Where the compiled graph spends its time

  • 64.8 ms per batch is the strict graph's eager overhead over the extended graph; 73.3% falls in Score and Mix, and softmax accounts for 9.8 ms.The eager module totals are 117.0 versus 52.2 ms per batch, a 2.2 ratio.Source: S4. Where the compiled graph spends… · p. 32
  • Routing reduces strict-graph Score time from 50 to 11 ms per batch and working memory from 469 to 77 MiB, while centroid selection adds 9.6 ms.The routed Hadamard Score falls to 9.2 ms and 78 MiB, with 9.7 ms for centroid selection and a 65.2 ms module total.Source: S4. Where the compiled graph spends… · p. 32
  • 31.6 ms per batch is the Hadamard graph's eager Score time versus 49.7 ms for strict, with a 78.2 ms module total.Hadamard Score uses 471 MiB; strict holds a tensor of the same size.Source: S4. Where the compiled graph spends… · p. 32
  • The eager profile uses one kernel per node and synchronizes around modules, whereas torch.compile fuses elementwise chains in the compiled forward pass.The reported compiled fused kernels keep chain intermediates in fp32; accuracy results come from eager bf16 runs.Source: S4. Where the compiled graph spends… · pp. 31–32
  • At k=16, compiled forward passes take 2.20 ms per batch for the reference and 7.21, 9.18, and 11.08 ms for the extended, Hadamard, and strict graphs.The table covers compiled forward passes on batches already on GPU; the eager profile instead times top-level modules across 12 blocks.Source: S4. Where the compiled graph spends… · p. 32, Table S2 · p. 32

S5. Polarization and squares

The section compares two exact square-based identities for replacing activation products and examines their computation, rounding behavior, and quantization implications. Shared factor squares make the norm expansion cheaper in contractions, while operation order and operand scaling affect bf16 error.

S5. Polarization and squares

  • Polarization uses two squares per isolated product; norm expansion uses three when factor squares cannot be shared.Source: S5. Polarization and squares · p. 32
  • The norm expansion costs N^2d + 2Nd squares for QK⊤, versus 2N^2d for polarization, because factor squares are shared across products.The strict graph uses the norm expansion where factor squares are shared and polarization elsewhere.Source: S5. Polarization and squares · p. 33
  • Both identities are exact in real arithmetic and in integer arithmetic without overflow or requantization.Source: S5. Polarization and squares · p. 33
  • 57% relative error results from rounding partial sums before subtraction, while subtracting close squares per term reduces error to 14% for either identity.The value-mix cancellation ratio is 65 at k = 16 and 543 with all keys on DeiT-Tiny.Source: S5. Polarization and squares · p. 33
  • Scaling the operands reduces value-mix error to 2.7% using λ² = ∥v∥/∥p∥.The fixed affine maps p → λp and v → v/λ preserve the product.Source: S5. Polarization and squares · p. 33
  • PTQ calibration ranges determine a scaling choice that balances operand ranges and can be folded into fixed affine maps at no runtime cost.The section also notes that the strict graph uses 12-bit input codes for fixed affine nodes, and Brent’s scaling can reduce this bit width.Source: S5. Polarization and squares · p. 33

S6. Tail-mass ablation on DeiT-Tiny

The ablation compares calibrated, exact, and conditional-mean estimates of omitted attention mass. Conditional-mean tail mass has no greater expected squared error than a constant estimate, while exact per-query mass varies substantially across queries.

S6. Tail-mass ablation on DeiT-Tiny

  • Calibrated and exact tail-mass means agree within 0.012, and within 0.001 for k ≥ 16, although individual queries vary widely.Calibration uses a per-block constant from training; exact mass is per query on the held-out Pet test split.Source: S6. Tail-mass ablation on DeiT-Tiny · p. 34, Table S5 · p. 34
  • The conditional mean E[τ | ϕ] has expected squared error no greater than the constant estimate, for statistics ϕ such as kept-score log-sum-exp, the kth score, or dropped-key count.Source: S6. Tail-mass ablation on DeiT-Tiny · p. 34
  • Exact mode uses q = τ, calibrated mode uses q = Eτ, and predicting dropped partition sum as m g(ϕ) yields q = 0 when no key is dropped.The latter formulation also covers causal attention, where tail mass grows with query position.Source: S6. Tail-mass ablation on DeiT-Tiny · p. 34
  • Table S4 compares top-1 accuracy on DeiT-Tiny/Pet across tail modes, including the saved one of five Jeffreys-distilled runs with trainable q.Source: Table S4 · p. 34

S7. Distillation loss

On DeiT-Tiny/Pet, Jeffreys distillation recovers a larger share of the accuracy gap than cross-entropy for the tested top-k settings. Its largest gains occur at k = 2 and 4.

S7. Distillation loss

  • For 2 ≤ k ≤ 32, Jeffreys distillation recovers 45% to 46% of the accuracy gap, versus 36% to 43% for cross-entropy.Table S6 reports top-1 accuracy for the saved run and mean ± s.d. over five seeds.Source: S7. Distillation loss · p. 35, Table S6 · p. 35
  • Jeffreys distillation gains 1.2 to 1.5 top-1 accuracy points at k = 2 and 4, where it matches the full teacher distribution rather than only the label.Source: S7. Distillation loss · p. 35

S8. bf16 against fp32

The fp32 compiled graphs reproduce reference results across the reported tasks, while bf16 has limited impact on DeiT-Tiny and MambaIRv2-Light. Pythia-70M is evaluated in fp32 because bf16 logits lose token ranking.

S8. bf16 against fp32

  • DeiT-Tiny and Pythia-70M results repeat exactly between runs, while MambaIRv2-Light runs differ by up to 0.01 dB.Source: S8. bf16 against fp32 · p. 35
  • In fp32, compiled graphs with exact GELU reproduce their references; on Pet, extended, strict, and Hadamard graphs match DeiT-Tiny accuracy and loss to four digits.The extended graph matches on ImageNet-1k and WikiText-2 and on Set5 within run-to-run variation.Source: S8. bf16 against fp32 · p. 35
  • bf16 changes DeiT-Tiny accuracy by at most 0.46 points and MambaIRv2-Light PSNR by at most 0.04 dB.Source: S8. bf16 against fp32 · p. 35
  • Pythia-70M’s bf16 validation NLL rises from 4.55 to 6.68 nats for both reference and compiled graphs, while bf16 weights with fp32 arithmetic give 4.56.The output projection accounts for 1.7 of the 2.1 nats of degradation and attention for the other 0.4; bf16 steps of 8 near logits around 1000 lose token ranking.Source: S8. bf16 against fp32 · p. 35

S9. Quantization with exact GELU

Post-training quantization costs the strict graph less accuracy than the extended graph, while the strict graph’s unfused computation has substantially greater intermediate-value traffic than native QK⊤. The section also reports graph node counts and the value-traffic reduction from top-k mixing.

S9. Quantization with exact GELU

  • At most 1.5 points versus 2.5 to 3.9 points: PTQ costs the strict graph less than the extended graph with either GELU.Table S8 compares strict graphs with exact GELU and x σ(1.702x) before distillation; Table 3 reports deployment configurations, including the extended graph after Jeffreys distillation.Source: S9. Quantization with exact GELU · pp. 35–36, Table S8 · p. 36
  • 344 nodes with GELU and 380 with its approximation, or 332 and 368 at k = 197, are reported for the quantized graph.Exact GELU matches no NPU configuration and is included as a comparison; the approximation is x σ(1.702x).Source: S9. Quantization with exact GELU · pp. 35–36

S10. Tensor sizes

  • 2.48 × 10^6 versus 3.88 × 10^4 values per head per image: the strict graph writes d times as many values as native QK⊤ for DeiT-Tiny.The unfused strict graph writes q_if + k_jf sums, squares them in place, and reduces over f; d = 64 drives the memory column of Table 2.Source: S10. Tensor sizes · p. 36
  • Top-k mixing reads k of N value rows per query, or k/N of dense mixing’s value traffic.The reported table context also identifies bf16-versus-fp32 comparisons with all keys and the PTQ table’s accuracy and change metrics.Source: S10. Tensor sizes · p. 36, Table S7 · p. 36, Table S8 · p. 36
Loading 2610.28958v1…