What actually runs: a measurement study of language model placement and decode speed on the Apple Neural Engine

Shahir M A

Source-linked AI summary

What actually runs: a measurement study of language model placement and decode speed on the Apple Neural Engine

Shahir M A

arXiv:2608.22110v1Aug 22, 2026cs.LGcs.ARcs.PF

TL;DR

The paper examines what determines language-model placement on and decode speed from the Apple Neural Engine. Using measurements of formulation, trained models, and actual memory traffic, it finds that encoding and operator choice govern accelerator residency and speed, motivating an encoding-first design procedure.

Problem
Prior work did not measure operator placement and decode speed on the Apple Neural Engine with weight encoding treated as a design variable.
Method
The study combines a formulation-controlled primitive matrix, matched trained models across size and precision, and ANE memory-controller byte counters during inference.
Results
Placement depends on architecture, size, and encoding; once resident, decode streams the entire weight set each token, making encoding the exchange rate between parameters and speed.
Takeaways & Limitations
Choose the encoding first, then allocate the resulting byte budget to parameters; grouped attention provides an additional operator lever on this accelerator.
Takeaways & Limitations
The placement boundary is specific to M1, macOS 26.1, coremltools 9.0, and conv-heavy architectures, and was not mapped for other model families.

Abstract

from arXiv · show

We ask what gets a language model onto the Apple Neural Engine (ANE) and what makes it fast there, and we answer with three measurements. We sweep a 64-shape matrix of LLM primitives that varies how a computation is expressed while holding what it computes fixed, recording per-operation device support. We then train matched models across size and precision, with quantized checkpoints byte-identical in structure to their fp16 counterparts, so every deployment measurement is of a real trained artifact. And we read the ANE's memory-controller byte counters during inference, establishing what actually ran rather than what the compiler intended. We support every headline claim with at least two of these three measurement paths. We find that placement is a property of how a computation is expressed, not of what it computes: a fused RMSNorm is fully ANE-eligible while its arithmetically identical decomposition is CPU-only. Weight encoding gates the accelerator: CoreML assigns a 25.85M-parameter conv-heavy fp16 model entirely to the CPU (our counters confirm zero bytes through the engine), while the same graph in int8 or 2-bit returns to ~83% residency and runs 1.8-2.2x faster, and a smaller 22.29M all-attention fp16 model sits at 98.9%. Decode cost is bytes streamed per token, at a constant ~0.77 fraction of nominal encoding width across fp16, int8 and 2-bit. The smallest and fastest models we measured are ternary, and at matched size the operator mix barely moves either axis: every resident 25M ternary model lands within 10.0-10.8 MB and 0.62-0.64 ms/token. The headline pair is half-attention ternary at 25M (10.5 MB, 0.63 ms) and 50M (16.8 MB, 0.86 ms) - 9.8x and 6.1x smaller, 3.0x and 2.2x faster than the conv-heavy fp16 design this work began with. From these measurements we draw a design procedure: choose the encoding first, then spend the byte budget on parameters.

1. Introduction

This study measures what determines language-model placement and decode speed on the Apple Neural Engine, addressing questions prior work and available tooling leave unresolved. Its evidence combines controlled operator sweeps, matched trained models, and runtime byte counters, leading to an encoding-first design procedure.

Prior work does not measure ANE operator placement and decode speed with weight encoding treated as a design variable.
The study uses three independent measurements: a 64-shape primitive matrix, matched trained models across size and precision, and ANE memory-controller counters.The measurements target supported devices, deployment comparisons, and actual runtime execution, respectively.
Every headline finding is supported by at least two independent measurement paths.
Placement depends jointly on architecture, size, and encoding, and must be measured rather than inferred from compiler intent.
Once resident, decoding streams the entire weight set each token, making encoding the exchange rate between parameter count and speed.
The resulting procedure is to choose encoding first, then spend the byte budget on parameters rather than quantizing only after training.
Accuracy costs are outside this study: the authors measure placement and speed while leaving encoding and operator-mix quality to a companion study.

2. Background: architecture is a function of the device

On-device language-model architecture must be designed for the target hardware because FLOPs, parameters, and even operator rankings do not reliably predict latency. Prior work established hardware-specific search, but this study targets the Apple Neural Engine, where encoding can determine whether a model runs on the accelerator at all.

Prior evidence: FLOPs and parameter counts can mispredict measured latency across devices and hardware backends.Prior studies report weak cross-device cost correlations and cases where similar FLOPs produce substantially different latency.
Why decoders differ: The relevant decoder memory is dominated by weights, unlike vision settings where activation memory can shrink with resolution or depth.This makes weight encoding a direct design variable for ANE language-model deployment.
Research gap: Hardware-aware architecture search has targeted many platforms, but prior work did not measure language-model placement and speed on the Apple Neural Engine.The gap includes accelerator-side measurements of operator placement, streamed bytes, and latency through the public deployment path.
Claim: On this device, choosing the wrong encoding can prevent accelerator execution rather than merely create a latency trade-off.The paper frames this as the sharper consequence of hardware-dependent architecture design on the ANE.
Target device: The study measures the ANE because it is an always-available approximately 1–2 W accelerator across two billion Apple devices.It is presented as a likely on-device language-model execution target within a power budget.

3. Method overview

The paper uses three complementary measurement stages: test compiler eligibility on controlled primitive expressions, deploy trained models, and verify execution with hardware counters. The matrix isolates expression-dependent placement, while the trained-model and counter measurements establish practical deployment behavior and its observability limits.

Method overview: The study first tests compiler acceptance on toy single-primitive models, then exports genuine trained models and measures their hardware execution.This ordering separates representability from behavior on deployable artifacts.
4.1 How we built the matrix: A 64-shape matrix holds semantics fixed while varying primitive formulation, shape, and expression to measure per-operation ANE eligibility.Six RMSNorm formulations provide a controlled comparison of equivalent computations, while negative controls test whether zero readings reflect real exclusions.
4.1 How we built the matrix: 63 of 64 tested shapes compiled; the int64 embedding gather failed to compile, while 49 shapes were fully ANE-eligible.The matrix covers attention, normalization, activations, linears, RoPE, LoRA, and short convolution families.
4.2 What the matrix shows: The matrix identifies actionable exclusions: embedding gather is CPU-only, convolution falls off between kernels 12 and 15, and dynamic-shape graphs can fall back silently.Eligibility and device preference are reported separately because eligibility alone does not show whether a tiny operation is favored by the ANE.
4.2 What the matrix shows: Placement depends on expression: fused RMSNorm is 100% ANE-eligible, whereas the arithmetically identical subln decomposition is 0% and CPU-only.Every constituent operation in the decomposition reports CPU as its only supported device.

5. Stage two: training matched models and measuring them on the hardware

Stage two trains matched models across size and precision, exports them through the deployment path, and measures latency alongside ANE byte traffic. The hardware counters validate whether weights moved through the engine, while chip-specific telemetry handling defines important reproduction boundaries.

5.1 The training harness: Matched trained models vary only size and precision under a fixed training setup, ensuring deployment measurements use real checkpoints rather than untrained artifacts.The harness fixes data, vocabulary, block size, training tokens, learning rate schedule, and seed while varying the specified arms.
5.2 Reading the ANE's byte counters: ANE memory-controller byte counters are read during continuous inference to measure bytes per inference and effective bandwidth.Bytes are preferred to power because they indicate whether the engine streamed weights and scale directly with elapsed time to yield bandwidth.
5.2 Reading the ANE's byte counters: The compute-unit control reads exactly zero when the engine is excluded, corroborating that nonzero counter activity reflects ANE execution.The study also uses CPU-only latency and CPU rail power as independent controls.
5.2 Reading the ANE's byte counters: The counter signal tracks nominal encoding width at a constant factor and agrees with latency changes, supporting aggregate weight-traffic interpretation.The method establishes aggregate traffic, not per-operation attribution.
Reproduction boundary: ANE telemetry is chip-specific: channel groups, names, units, and availability differ across M1 and M3, so name-only matching can silently produce confident zeros.Matching must use group, name, and unit jointly, and channels must be enumerated on the target chip before measurement.
Measurement configuration: Latency is reported as best-of-30 warm predicts under CPU_AND_NE beside a CPU_ONLY control, so these measurements are warm figures rather than sustained throughput.Training records are regenerated from an append-only ledger to keep reported configurations tied to the producing runs.

6. Encoding gates the accelerator

On the ANE, placement depends jointly on architecture, size, and encoding, so it must be measured rather than inferred from computation alone. Encoding can move a small model from CPU-only execution to substantial ANE residency, while operator family changes the placement boundary.

6.1 A size cliff, within one architecture family: 0% ANE residency occurs for the 25.85M-parameter conv-heavy fp16 model, while the 48.53M fp16 model reaches 98.2%.A width-and-depth sweep places the conv-family boundary between 25.9M and 31.1M non-embedding parameters.
6.3 The boundary is architectural, not a parameter count: 98.9% residency is achieved by the smaller 22.29M all-attention fp16 model, showing that the conv-family boundary is not a parameter-count rule.The attention model streams 36.6 MB/token through the engine, whereas the larger 25.85M conv model moves exactly zero bytes through it.
6.4 Reconciling the two directions: Increasing model work and decreasing encoded bytes move placement in the same direction by changing the balance between CPU and ANE costs.The ANE's dispatch overhead penalizes small graphs, while quantization reduces bytes and transfer cost.
6.5 The placement result reproduces on a second chip: 1.76× faster execution on M3 for int8 versus fp16 confirms encoding-gated placement across two chip generations and two macOS versions.The fp16 model draws 0 mJ of ANE energy, while int8 draws approximately 2,200 mJ; CPU controls independently confirm the placement difference.

7. Speed: decode streams the whole weight set every token

Decode is dominated by streaming the model's weights once per token, making encoded bytes the key determinant of on-device speed. Measured byte traffic tracks nominal encoding width with reuse, while fixed dispatch costs remain visible at small scale.

Weight traffic: Every decode token performs one full pass over the weight set, placing short-context batch-1 inference on the parameter-bound side of the latency trade-off.The paper reports that KV caching buys nothing in this regime because per-token cost is the weight pass rather than context.
Weight traffic: 0.77 is the approximately constant fraction of nominal encoding width observed in measured bytes per parameter per token across encodings.The reuse factor is attributed to the SRAM working set.
Encoding and latency: Encoding converts parameters into speed at a fixed exchange rate because decode latency is governed by streamed bytes.The paper characterizes inference as a data-movement problem before a compute problem.
Measured deployment speed: Resident models align on a single latency-versus-package-size line, whereas CPU-kept models form the comparison set for the deployment measurements.Figure 4 plots latency against deployed size for eleven trained models.
Dispatch overhead: 2.31 ms for 128-token prefill is barely above one decoded token at 1.90 ms in fp16, exposing a fixed per-dispatch cost at this scale.The public-path floor is approximately 0.54 ms per inference.

8. The other lever: operator choice

Operator mix affects ANE placement more than resident ternary decode speed. Grouped attention can be smaller than the convolutional block and more likely to clear fp16 placement, while matched resident ternary mixes have nearly identical device-axis measurements.

Why attention clears fp16 placement: 2.5d² makes grouped-GQA attention smaller than the convolution block's approximately 4d² projection cost at two KV heads.The advantage shrinks as the number of KV heads approaches the number of query heads.
Matched ternary models: 0.618–0.639 ms and 10.0–10.8 MB span the three resident 25M ternary operator mixes, showing little speed or size variation.At 50M, the corresponding latencies are 0.865 and 0.859 ms.
Headline models: 9.8× smaller and 3.0× faster at 25M, and 6.1× smaller and 2.2× faster at 50M, the half-attention ternary models outperform the starting conv-heavy fp16 design on device axes.The 25M model measures 10.5 MB and 0.63 ms; the 50M model measures 16.8 MB and 0.86 ms.
Scope: Operator choice remains unresolved on quality: matched ternary mixes tie on device axes, so accuracy determines the choice outside this paper's scope.The paper does not pursue which operator mix is most accurate.

9. Putting it together: a design procedure

The paper's procedure is to choose encoding before parameter count, convert the byte budget into parameters, and treat operator mix as placement-critical at fp16 but nearly free on resident ternary device axes.

Design procedure: 1. Choose the encoding before the parameter count because it gates placement below approximately 30M parameters.Below that size, a conv-heavy fp16 model does not reach the accelerator.
Design procedure: 2. Convert the byte budget into parameters because decode cost is bytes per token; at fixed latency, int8 buys approximately 2× and 2-bit approximately 8× fp16 parameters.
Design procedure: 3. Treat operator mix as free on resident ternary device axes but placement-critical at fp16.Grouped-GQA attention is smaller than the LIV convolutional block and is the mix CoreML admits at fp16.
Design procedure: 4. Express operations in ANE-native form, including fused RMSNorm, fixed shapes, CPU embeddings, and convolution kernels of at most 12.

10. Limitations

The study’s conclusions are bounded by measurement scope, platform configuration, and the absence of quality evaluation. Several findings are M1-specific, while mobile thermal behavior and some cross-device byte analyses remain unresolved.

Quality is not measured in this work; the accuracy question is deferred to a companion study using the same artifacts.
Placement and size-boundary claims are specific to CoreML on M1/macOS 26.1/coremltools 9.0 and conv-heavy architectures, although encoding-gated placement reproduces on M3.
Cross-chip evidence is limited to single-run M3 energy measurements, because M3 lacks the byte counters needed for streaming analysis.
The int8 and 2-bit streaming findings conflict with a published M1 account, but the present data cannot separate runtime-path differences from compiler-version differences and do not claim refutation.
Throughput is measured as best-of-30 warm on a Mac; sustained phone throughput remains unmeasured despite a 4.6% peak-to-final decay during a five-minute M1 ANE check.
The counters establish aggregate engine activity rather than per-operation attribution, so residency is corroboration of the compute plan, not exact operator-level accounting.

11. Relation to prior work

The paper builds on established ANE, roofline, quantization, and NPU literature while contributing a reusable public-path measurement procedure. It distinguishes its measurements from prior private-runtime and published M1 accounts.

Prior work establishes the ANE datapath, roofline, weight-compression scheme, dispatch floor, static-shape rigidity, and sub-4-bit LUT-only behavior.
Other-vendor evidence already reports fp16 as catastrophically slower than int8 on NPUs, placing this study’s encoding comparison in a broader context.
The paper contributes a reusable measurement procedure for operator placement, streamed bytes, and latency through the public deployment path.

12. Reproducibility and availability

The authors make the measurement pipeline reproducible through ledger-derived paper generation, public code and harnesses, and documented export and runtime procedures. They also provide a separate M3 cross-device report.

Model-configuration tables are regenerated from the append-only results/ledger.jsonl run ledger via paper/build_paper.py.
Code, ledgers, and measurement harnesses are publicly available at the project’s GitHub repository.
The documented pipeline covers quantizer tests, data preparation, checkpoint harvesting, ANE export, residency probing, byte-counter compilation, and ledger regeneration.
The M3 cross-device arm was run separately, with raw and normalized numbers stored in results/2026-07-30_m3_cross_device/.
CoreML export requires Python 3.13 and coremltools 9.0 with native libraries, plus fixes for causal-mask tracing, GQA broadcast, and compiled-model loading.