Pipeline-Native Transformers: Co-Designing Model Architecture and CPU Inference for Bandwidth-Efficient Autoregressive Decode

Note on the Use of AI Tools

• Anthropic’s Claude assisted with cflow implementation and report drafting and editing, while the author directed the research and accepts responsibility for its contents.The author states that architectural and experimental design, analysis, conclusions, and responsibility for errors remained their own.Source: Note on the Use of AI Tools · p. 3

1.1 The Bandwidth Bottleneck in LLM Inference

• Single-token CPU decoding is bandwidth-bound because roughly 1 TFLOP/s of arithmetic and 50 GB/s of memory bandwidth yield a 20 FLOP/byte machine balance.Q4 matrix-vector products reach only 4 FLOP/byte, so latency is governed primarily by bytes transferred rather than operations executed.Source: §1.1 · p. 11

1.2 The GPU-First Retrofit Problem

• GPU-derived CPU runtimes retain mismatched weight layouts, execution order, and expert loading that defeat prefetching and can load 16× more expert bytes than selected.For Gemma 4’s 128 experts with top-8 selection, loading all experts creates a 15-fold avoidable bandwidth overhead.Source: §1.2 · p. 12

1.3 The Co-Design Opportunity

• Co-designing architecture and runtime around the CPU memory hierarchy enables bandwidth reductions unavailable to either independently, using cflow tiles and pipeline-native dependency graphs.cflow reads L2-sized tiles in consumption order, while pipeline-native architectures make vertical stage-major execution mathematically valid.Source: §1.3 · p. 13

1.4 Contributions

• The report contributes cflow, dependency-relaxation operations, five trained architectures, and a delay-aware scheduler that matches PyTorch traces within 0.006% relative norm error.The scheduler uses dense_delay and expert_delay parameters, ring-buffered residual history, and delayed expert injection.Source: §1.4 · pp. 13–14

1.5 Summary of Key Results

• cflow sustains 5.94 tok/s on a 30.9B-parameter pipeline-native MoE, ahead of llama.cpp’s 4.75 tok/s on a parameter-comparable dense model on the same CPU.Source: §1.5 · p. 14
• 2.00×: arch2_4_combined achieves the largest bandwidth reduction, while arch4_async_experts achieves the best perplexity at 6.26 versus 6.50.The results identify dense_delay as the bandwidth knob and expert_delay as the quality knob, placing the two architectures at opposite trade-off extremes.Source: §1.5 · p. 15

1.6 Scope and Limitations

• The evidence is limited to single-token decode and proof-of-concept TinyStories models; batched inference and state-of-the-art language-model quality are not claimed.Claims 6 and 8 are explicitly unproven: storage-to-memory I/O dominates at 4.7 GB, preventing PREFETCHT0 or stage-major overlap from helping.Source: §1.6 · pp. 15–16

1.7 Report Organization

• The report proceeds from CPU and transformer background through cflow, pipeline-native architectures, evaluation, discussion, and conclusion.The evaluation covers training quality, cache locality, bandwidth, storage I/O, negative results, and Rust–Python parity.Source: §1.7 · p. 16

2.1.1 The Memory Hierarchy

• 40–50 GB/s of sustained main-memory bandwidth, versus roughly 400 GB/s in L1, creates an 8–10× gap that tile streaming targets.The representative hierarchy lists 32 KB L1 at ≈400 GB/s, 256 KB L2 at ≈200 GB/s, shared 20 MB L3 at ≈100 GB/s, and DDR3 at ≈40–50 GB/s.Source: §2.1.1 · p. 17

2.1.2 The Roofline Model

• 4 FLOP/byte is five times below the 20 FLOP/byte machine balance, making single-token transformer decode memory-bandwidth-bound rather than compute-bound.In Q4, each weight costs 0.5 bytes, and each matrix-vector product performs 2mn FLOPs while loading 0.5mn bytes; weights are loaded once per token.Source: §2.1.3 · p. 18, §2.1.2 · p. 18

2.2.1 Architecture

• The pre-norm decoder processes attention and FFN residual updates layer by layer, while autoregressive inference reuses prior-position keys and values through the KV cache.Attention loads Q, K, V, and output projections for the new token; GQA reduces KV-cache and KV-projection bandwidth by sharing key-value heads.Source: §2.2.1 · p. 19, §2.2.2 · pp. 19–20

2.2.3 Feed-Forward Network

• Q4 decode is dominated by the three GeGLU matrices, costing 3×d×dff×0.5 bytes per token, while fused AVX2 dequantization preserves the 0.5-byte-per-parameter bandwidth rate.Dequantized values flow directly through the arithmetic pipeline and are never materialized in memory.Source: §2.2.3 · p. 20, §2.3.1 · p. 21

2.2.4 Mixture-of-Experts Feed-Forward Layers

• Selecting k of E experts makes naive MoE execution waste (E−k)/k of expert bandwidth, whereas reading only selected tiles eliminates that waste.For Gemma 4 26B-A4B’s E=128 and k=8, naive execution moves 16× more expert weight bytes than it consumes; conditional top-8 loading is approximately 23.7 MB versus 379 MB per layer.Source: §2.2.4 · p. 20, §2.5 · p. 23

2.2.5 The Layer Dependency DAG

• Standard layers form a strict dependency chain because layer ℓ+1 cannot begin until layer ℓ completes both attention and FFN residuals, preventing stage-major decode.The background uses this dependency structure to motivate rewriting the inter-layer graph for vertical pipelining.Source: §2.2.5 · pp. 20–21

2.3.2 GGUF and llama.cpp

• cflow replaces GGUF’s row-major organization with compute-order 128×256 tiles, a separate random-access expert bank, and architecture-specific delay fields for schedule construction.Existing runtimes generally retain row-major, per-layer execution; llama.cpp also loads all MoE experts before discarding unselected ones.Source: §2.3.2 · p. 21, §2.4 · p. 22

2.6.5 Model-Runtime Co-Design

• The report’s co-design contribution rewrites transformer dependencies so vertical stage-major pipelining becomes valid, generalizing parallel attention–FFN structure with dense and expert delays.Unlike prior hardware-aware architecture search, this approach co-designs the rewritten dependency graph with the runtime schedule that exploits it; it accepts routing and optimizes its CPU bandwidth cost.Source: §2.6.5 · p. 24, §2.6.3 · p. 24

The cflow Runtime

• cflow combines L2-sized compute-order tiles, .cflow/.vflow storage, fused AVX2 attention and normalization, and conditional expert prefetch into a CPU-first streaming engine.The runtime is built around four primary components: tiled weights, two on-disk formats, a fused compute pipeline, and selective expert loading.Source: The cflow Runtime · p. 25

3.1 Design Principles

• Compute-order layout and zero-copy mmap let sequential reads feed OS readahead while mapped tile references are consumed directly without deserialization or heap allocation.Expert tiles remain separate because their data-dependent access pattern requires selective random access.Source: §3.1 · p. 25, §3.2.3 · p. 27
• Tiles are 128×256 submatrices, with approximately 18 KB per Q4 tile, allowing 8–12 tiles to fit in 256 KB of per-core L2 alongside the activation.This sizing keeps the hot activation in L1 while tiles are processed from L2, avoiding RAM round-trips during each tile computation.Source: §3.1 · pp. 25–26, §3.2.1 · p. 26

3.3.1 GlobalHeader

• Runtime execution semantics come from serialized header fields for dense delay, expert delay, combination style, feature flags, and logit soft-cap rather than hard-coded model-family fallbacks.The v5 header records the parameters consumed by the delay-aware scheduler and architecture-specific execution paths.Source: §3.3.1 · pp. 27–28

3.3.3 Expert Offset Table and Conditional Loading

• 93.75% of expert weight reads are eliminated for E = 128 and k = 8 by selecting only router-chosen experts through an offset table.The runtime reads only the selected k experts after router scoring instead of loading all experts.Source: §3.3.3 · p. 28

3.4 The .vflow File Format

• The .vflow layout streams all non-expert attention and dense-FFN tiles for each vertical group sequentially, avoiding seeks while leaving expert tiles in a random-access bank.For Gemma 4, groups adapt to its 5:1 sliding/full-attention pattern; uniform-attention architectures use one group spanning all layers.Source: §3.4 · p. 28, §3.4.2 · p. 29, §3.4.1 · p. 29

3.5.2 Fused QKV and Gate+Up

• Fused QKV and gate-up tile streams reuse one activation load across multiple projections, reducing repeated activation traffic during tiled matrix-vector products.Q, K, and V tiles are interleaved by output stripe, and the same pattern is used for dense FFN gate and up projections.Source: §3.5.2 · p. 30, §3.5.1 · p. 30

3.6.1 Q4_0 Quantization Format

• 4-bit dequantization is fused into dot products, with runtime-detected AVX2/FMA, scalar fallback, and a separate AVX1+SSE4.1 path for older hardware.The AVX2 path processes 32 weights in approximately 10 instruction slots and is fully utilized on the Ryzen 5 2600; Sandy Bridge uses AVX1+SSE4.1.Source: §3.6.1 · p. 30, §3.6.3 · p. 31

3.8 Normalization Layers

• Normalization metadata is stored with tile data in the layer offset table and loaded fully at each layer, remaining resident in L2 throughout computation.The metadata is small enough to stay in L2 cache for the layer’s computation.Source: §3.8 · p. 33

3.9.1 Two-Thread Pipeline

• cflow uses separate compute and prefetch threads connected by a bounded PrefetchCommand channel, with hardware prefetch instructions pulling upcoming weights toward L1-d.The compute thread drives the forward pass and the prefetch thread issues _mm_prefetch on x86-64.Source: §3.9.1 · p. 33

3.9.2 Linear and Conditional Commands

• Conditional prefetching targets only the k experts predicted for the next layer, avoiding unused expert cache data while exploiting correlated routing decisions and incurring negligible per-layer channel overhead.The compute thread issues ExpertTiles after routing, while LinearRange commands prefetch attention and dense FFN regions one layer ahead.Source: §3.9.2 · pp. 33–34

3.9.3 Negative Result: PREFETCHT0 at RAM-Bottleneck Scale

• At 64 MB and 4.7 GB scales, PF=1 and PF=0 produce bandwidths within measurement noise, so PREFETCHT0 adds no measurable benefit under storage-to-RAM bottlenecks.The claim remains untestable until compute time per token significantly exceeds I/O time.Source: §3.9.3 · p. 34

3.9.4 Staged Direct-I/O Expert Fetch

• The runtime stages router-selected expert regions through direct-I/O reader threads, overlapping reads with intervening computation under expert_delay while preserving bit-identical deferred arithmetic.Per-slot pending-operation counters prevent buffer reuse during in-flight reads, and saved router activations support deferred computation.Source: §3.9.4 · p. 34

3.10.1 Architecture Dispatch

• DelayedMoESource orders dense and expert tiles by their delayed consumption positions, preserving sequential weight reads under architecture-specific dense_delay and expert_delay schedules.Architecture dispatch serializes parameters such as dense_delay, expert_delay, and combine_style in the GlobalHeader.Source: §3.10.1 · pp. 34–35, §3.10.2 · p. 35

3.10.3 Tile Quantization

• During conversion, cflow quantizes checkpoint tiles to per-group Q4_0, using 32-weight groups with one scale factor and f16-prefixed packed nibble data.The scale is s = max |w_i|/7, and quantized values are clipped to [0, 15].Source: §3.10.3 · p. 35

3.11 Implementation Correctness

• All 116 unit tests and 8 integration tests pass, with Rust↔PyTorch parity, format round-trip checks, and exact or near-exact outputs establishing correctness for performance evaluation.For arch2_4_combined, residual norms remain within <1% relative error, output argmax matches exactly, and top-32 overlap is ≥27/32; arch4_async_experts matches with <0.007% relative norm error and exact argmax.Source: §3.11 · pp. 35–36

Pipeline-Native Transformer Architectures

• The architectures permit a vertical pipeline schedule by construction, targeting total layer-stack weight traffic rather than only within-layer cache locality.This complements cflow’s cache-optimal weight streaming and conditional expert loading by reducing bytes read between successive token outputs.Source: Pipeline-Native Transformer… · p. 37

4.1.1 Formal Statement

• Standard pre-norm layers form a strict sequential chain because layer ℓ+1 depends on both mℓ and fℓ from layer ℓ.Each layer’s weights must be fully read before the next layer begins, so weight reads cannot be reordered without violating dependencies.Source: §4.1.1 · pp. 37–38

4.1.2 The Vertical Pipeline Idea

• A vertical schedule overlaps stages across layers, but it is mathematically invalid for standard transformers because layer ℓ+1 attention depends on layer ℓ’s FFN output.The proposed overlap requires next-layer attention to avoid dependence on the preceding FFN contribution, which standard architecture does not satisfy.Source: §4.1.2 · p. 38

4.1.3 Bandwidth Implications

• 2.00× reduction lowers critical-path bandwidth from 9.00 to 4.50 MB/token for arch2_4_combined by delaying dense and expert reads across layers.The 16.5 MB fully serial total is not the latency-setting quantity because attention, dense-FFN, and expert reads within each layer issue concurrently.Source: §4.1.3 · p. 39

4.2.1 The Dense Delay Transform

• Dense delay makes each FFN consume a residual Δ layers old, breaking the intra-layer dependency so the next layer’s attention can start sooner.The scheduler stores dense_delay, builds a ring-buffered residual history, and selects each layer’s FFN residual snapshot.Source: §4.2.1 · p. 39

4.2.2 The Expert Delay Transform

• Expert delay enqueues outputs at layer ℓ and injects them δe layers later, allowing expert tiles to load while intervening layers compute.The expert_delay value is stored in the .cflow header and defines the overlap window between routing and injection.Source: §4.2.2 · pp. 39–40

4.2.3 The CombineStyle Variants

• CombineStyle encodes distinct forward-pass semantics: ParallelSqrt2 runs Dense FFN and MoE in parallel on the same pre-FFN normalized residual, sums and scales outputs by 1/√2, with no delays.DelayedSum uses δd = 1 and δe = 2; its dense FFN reads a delayed residual while the router fires from the current post-attention normalized residual.Source: §4.2.3 · p. 40

4.3 The Five Candidate Architectures

• All five candidates use the same TinyStories training configuration and nominal geometry, while architectures 4 and 5 add MoE or weight sharing that increases parameter counts.The shared setup uses 10K steps, AdamW, cosine decay, a 50K GPT-2 vocabulary, d = 512, L = 6, and dff = 2048.Source: §4.3 · p. 40

4.3.1 Arch1: Decoupled Residual Streams

• Arch1’s decoupled attention and FFN streams synchronize every three layers but achieve only 1.00× bandwidth reduction and test perplexity 7.21.Its independent streams provide a stage-major pipeline opportunity, yet δd = 0 and δe = 0 leave the critical path unchanged.Source: §4.3.1 · p. 41

4.3.2 Arch2_4_combined: Dense-and-Expert Delay

• 4.50 MB/token versus 9.00 MB/token gives arch2_4_combined a 2.00× critical-path bandwidth reduction, with test perplexity 6.50.Its dense delay δd = 1 removes an FFN blocking dependency, while expert delay δe = 2 loads expert tiles during the intervening computation window.Source: §4.3.2 · p. 41

4.3.3 Arch2_4_sync: Synchronous Variant

• Arch2_4_sync records test perplexity 6.52 versus 6.50 for the combined schedule, while retaining the same 4.50 MB/token critical path.Removing the expert queue therefore costs no measurable quality but forfeits the expert-side overlap window for later I/O hiding.Source: §4.3.3 · p. 41

4.3.4 Arch3: Pipeline Registers

• Arch3’s explicit attention and transformation registers achieve 1.00× bandwidth reduction and test perplexity 7.24, matching arch1 as a baseline-quality reference.Layer ℓ+1 attention reads attn_regℓ independently of layer ℓ’s FFN, while the FFN reads the prior layer’s complete output.Source: §4.3.4 · p. 42

4.3.5 Arch4: Asynchronous Experts

• Arch4 reaches the best test perplexity, 6.26, improving 0.24 over arch2_4_combined, but expert delay alone leaves critical-path bandwidth unchanged at 1.00× reduction.Its pre-dense routing injects expert outputs two layers later; dense bandwidth dominates at this geometry.Source: §4.3.5 · p. 42

4.3.6 Arch5: Fixed-Point Iteration with Weight Sharing

• Arch5 achieves test perplexity 6.77 with 14.06 MB/token measured critical-path bandwidth, but only ≈4.69 MB of unique on-disk weights.Two weight blocks are iterated three times, enabling temporal weight reuse while illustrating a parameter-efficiency trade-off.Source: §4.3.6 · pp. 42–43

4.4.1 Ring-Buffered Residual History

• The scheduler maintains max(δd, δe) + 1 residual vectors in a ring buffer and selects delayed entries by modulo indexing.Both delay integers are read from the .cflow header, so no model-family-specific scheduler logic is required.Source: §4.4.1 · p. 43

4.4.2 Multi-Layer Driver

• The multi-layer driver processes layers in order, retrieves delayed residuals, injects queued AsyncExperts outputs, invokes the layer executor, and updates residual history.DelayedSum queues are handled internally, while AsyncExperts outputs are routed to target layer ℓ + δe.Source: §4.4.2 · p. 43
• At d = 512 and δe = 2, the residual ring buffer and pending expert vectors add approximately 10 KB of working state.The overhead comprises (δd + 1) × d × 4 bytes for residual history plus at most δe vectors of size d × 4 bytes.Source: §4.4.2 · p. 43

4.5.1 Critical-Path Analysis

• Delayed scheduling moves dense and expert reads off the critical path by spreading each stream across δ+1 layers, reducing their per-layer contributions by the same factor.Dense FFN reads use δd, while selected expert tiles use δe; intra-layer streams execute concurrently, so the longest amortized stream determines the path.Source: §4.5.1 · p. 44
• 4.50 MB/token is the arch2_4_combined critical-path bandwidth, down 2.00x from the 9.00 MB/token naive baseline.With δd = 1 and δe = 2, dense and expert streams amortize to 0.75 and 0.25 MB per layer, respectively; the longest concurrent stream sets the 0.75 MB per-layer path across six layers.Source: §4.5.1 · p. 44
• The 9.00 MB/token naive baseline, not the 16.5 MB fully serial total, is the valid comparison because it already credits intra-layer overlap; both figures are analytic rather than runtime timings.The model computes critical-path bandwidth from geometry and delay integers alone.Source: §4.5.1 · p. 44

4.5.2 Dense Delay vs. Expert Delay as Complementary Knobs

• Dense delay has twice the per-layer bandwidth impact at the arch2_4 geometry, while expert delay becomes primary when larger expert dimensions make expert traffic dominant.At arch2_4, dense FFN traffic is 1.5 MB/layer versus 0.75 MB/layer for experts; at Gemma 4 scale with k = 8, the relationship reverses.Source: §4.5.2 · p. 45
• 0.24 perplexity improvement from arch4’s pre-dense routing sacrifices the dense bandwidth reduction, exposing a direct bandwidth–quality trade-off across candidates.Table 4.1 summarizes the five trained architectures, while Figure 4.1 plots their bandwidth–quality surface; arch2_4_combined and its sync ablation are the only points at 4.50 MB/token.Source: Figure 4.1 · p. 45, §4.5.2 · p. 45, Table 4.1 · p. 46

• 2.00× bandwidth reduction trades against quality: arch2_4_combined reaches ppl 6.50, whereas arch4 reaches ppl 6.26 but gives up dense bandwidth reduction.The trade-off is tunable through (δd, δe) and the routing anchor, with no single Pareto-optimal point.Source: §4.6 · p. 46
• ppl 6.77: arch5 achieves competitive quality with one-third of arch1/arch3’s parameter count, while its 14.06 MB bandwidth reflects six passes over two unique blocks.The unique on-disk bytes are 4.69 MB, making arch5 most efficient when storage, rather than per-token bandwidth, is the constraint.Source: §4.6 · p. 46
• Modest implementation costs support the bandwidth gains: CombineStyle adds one conditional branch per layer, while delay-aware scheduling uses a ring buffer and expert queue with O(δ · d) memory.The scheduler is parameterized by header fields rather than model-family hard-codes.Source: §4.6 · p. 46
• 2.00×: arch2_4_combined alone reduces critical-path bandwidth with δd = 1, δe = 2, while landing within 0.24 perplexity points of arch4’s best quality, ppl 6.26.The reduction is measured against the sequential schedule under CombineStyle::DelayedSum.Source: §4.6 · p. 46
• 403 MB, 134 MB, and 101 MB: the measured 31B model preserves the same 2.00× critical-path reduction at d = 8,192 as at d = 512.With dff = 4d, the dense stream remains 3× the attention stream; inherited Gemma 4 geometries instead fall below 1.50× when attention binds.Source: §4.6 · p. 47

5.2.2 Results

• All five architectures trained effectively, with perplexity ranging from 6.26 to 7.24; arch4 reached 6.26 while arch1 and arch3 reached 7.21 and 7.24.The results support effective gradient flow through delay paths, and the reported ordering remained stable across seed ablations.Source: §5.2.2 · pp. 49–50

5.2.3 Large-Scale Validation: arch2_4_8k_4l

• The 8.34B arch2_4_8k_4l run reached validation perplexity 4.52 and top-1 accuracy 61.4%, demonstrating quality and training stability at scale but not a delayed-versus-undelayed comparison.No undelayed control was trained at this geometry; the comparison exists only at the 114M scale.Source: §5.2.3 · p. 50

5.3 Claim 1: Conditional Expert Loading

• Conditional expert loading is structurally proven: reading k of E experts consumes exactly k/E of expert bytes, giving 25% for E = 8, k = 2 and 6.25% for E = 128, k = 8.The .cflow expert offset table lets the runtime issue reads only for router-selected experts.Source: §5.3 · p. 50

5.4.2 PMU Hardware Counter Measurement (Linux KVM, May 2026)

• The tiled layout produced 6.7–7.3× fewer L1-d read misses, including 2.64M versus 19.3M misses and a 7.29× improvement for dense-down.The same tiled implementation also achieved 1.12–1.36× wall-clock speedups across representative workloads, with the largest gain on attention projection.Source: §5.4.2 · pp. 51–52, §5.4.1 · p. 51

5.6 Claim 4: Fused Projections

• The file format and dispatch loop structurally guarantee activation-load savings of 3× for QKV and 2× for gate+up.The AVX2 path matched scalar outputs within 10−4 on 1,000 random tiles, and all 116 unit tests plus 8 integration tests passed with zero failures.Source: §5.5 · p. 52, §5.6 · p. 53

5.8.2 4.7 GB Scale (arch2_4_8k_4l, Windows SATA SSD)

• Explicit PREFETCHT0 was refuted: it changed direct-I/O bandwidth by at most ±0.1% at 64 MB and was approximately 4% worse at 4.7 GB.On the 30.9B model, disabling prefetch increased throughput from 2.58 to 4.96 tok/s, a 1.92× speedup, because storage bandwidth was the bottleneck and prefetch added cache pressure.Source: §5.8.1 · p. 53, §5.8.2 · p. 54

5.9 Claim 7: Vertical Pipeline Bandwidth Reduction

• 2.00× lower critical-path bandwidth was analytically validated for arch2_4_combined, while the overlap mechanism later achieved up to 1.68× wall-clock improvement.The reduction is unique among the five architectures; the others retain Bcritical = Bnaive because their delays are zero.Source: §5.9.2 · p. 55

5.10.2 4.7 GB Scale

• The stage-major .vflow readahead benefit remained inconclusive: at 64 MB it was approximately 1% slower than .cflow, and the larger-scale consistency difference was confounded by SSD cache state.The proposed benefit requires asynchronous overlapped streaming, which the current single-thread sequential executor does not implement.Source: §5.10.2 · pp. 55–56, §5.10.1 · p. 55

5.11.2 arch4_async_experts

• All 116 unit tests and 8 integration tests passed, while Rust replay matched the reference output argmax of 9760 on both .cflow and .vflow paths.The arch2_4_combined parity tests covered both paths, and arch4_async_experts replay reported relative norm error below 0.007%.Source: §5.11.3 · pp. 56–57, §5.11.2 · p. 56

5.12 Summary of Results

• 2.00×: arch2_4_combined reduces critical-path bandwidth, supported by a 7.29× reduction in L1-data read misses; six of eight thesis claims hold overall.The scorecard reports two negative results, each with conditions under which it could be revisited.Source: §5.12 · p. 57, Table 5.3 · p. 57

5.13.1 Benchmark Model and Hardware

• The end-to-end benchmark used arch2_4_8k_16l, a 30.9B, 17.39 GB Q4 model whose top-2 routing reads approximately 10.2 GB of weights per token on an Ice Lake server.The model has 16 layers, hidden size 8,192, eight experts, and two selected experts per layer.Source: §5.13.1 · p. 58

5.13.2 cflow Decode Optimization

• 2.30×: disabling harmful prefetching and repeated tile-range allocation lifted cflow throughput over the initial implementation, reaching 5.94 tok/s and 60.6 GB/s of weight movement.The prefetch thread consumed approximately 48% of decode time and disabling it alone produced a 1.92× speedup.Source: §5.13.2 · pp. 58–59, Table 5.4 · p. 58

5.13 Head-to-Head Comparison with llama.cpp and vLLM

• 5.94 tok/s: cflow outpaces llama.cpp at 4.75 and vLLM at 1.65 on the same CPU, but the comparison is not quality-controlled because the models differ.cflow uses a 30.9B top-2-of-8 MoE with approximately 20B active parameters per token; baselines use dense Qwen2.5-32B models.Source: §5.13.1 · p. 58, Table 5.5 · p. 59, §5.13.3 · pp. 59–60

5.14.1 Design

• Async expert reads preserved bit-identical greedy outputs while moving storage reads from routing to injection across a two-layer delay window.The staged arm issues reads asynchronously and awaits them after overlapping intervening computation; both arms read identical bytes and perform identical arithmetic.Source: §5.14.1 · p. 61

5.14.3 Part 2 — EC2 r6id.8xlarge (NVMe tier): the net win

• 1.68×: asynchronous expert reads matched the overlap model within 1% on r6id.8xlarge, whereas desktop SATA experiments hid 79–89% of stalls but produced no consistent net speedup.The refined model incorporates an interference tax on overlapped compute and an effective-I/O floor imposed by the storage tier.Source: §5.14.2 · p. 61, Figure 5.1 · p. 62, §5.14.3 · p. 62

5.14.4 Part 3 — Parallel fetch: a null that locates the ceiling

• Four parallel fetch workers changed nothing because the instance-store drive was bandwidth-capped, leaving approximately 512 ms/token of unavoidable I/O and a ceiling near 2 tok/s.The residual delay was attributed to pipeline fill/drain and the clamped last-layer fetch rather than fetch-side queue depth.Source: §5.14.4 · pp. 62–63

6.1.1 What Co-Design Means Here

• Pipeline-native co-design differs from weight offloading, sparsity, routing optimization, HW-NAS, and tiered streaming by rewriting the dependency graph, and it can complement those techniques.The report defines co-design narrowly as modifying inter-layer dependencies so vertical stage-major pipelining is logically valid; ordinary transformers cannot use that schedule without changing dependencies or approximating execution.Source: §6.1.1 · p. 64, §6.1.3 · pp. 65–66

6.1.2 The Scope of the Result

• 2.00× critical-path bandwidth reduction is analytically guaranteed for arch2_4_combined, but wall-clock benefit requires off-path reads to overlap and varies with model geometry.At training geometry the model fits in RAM, so no separate overlap window exists; disk-resident expert-tier experiments later measured up to 1.68× benefit, while Gemma-scale reduction requires separate calculation.Source: §6.1.2 · pp. 64–65

6.2.1 How the Bandwidth Reduction Scales

• The co-design helps when delayed dense-FFN or expert streams set the critical path; at Gemma 4 scale, expert amortization shifts binding to attention, leaving whole-model gains below 1.50×.At E = 128, k = 8, the expert stream is 23.7 MB/layer; with δe = 2, it amortises to 23.7/3 = 7.9 MB, below the 15.8 MB attention stream.Source: §6.2.1 · p. 67, §6.2.3 · p. 67

6.2.2 Attention-Path Delays and Future Work

• Attention-path delay could lower the remaining floor, but delayed residuals would stale the KV cache; speculative pipeline recovery and router-based FFN prediction remain untested future alternatives.Approach A is explicitly proposed rather than evaluated, while Approach B predicts FFN residuals from router logits and is viable only if the predictor achieves R2 > 0.7.Source: §6.2.2 · p. 67, §6.3 · p. 68, §6.3.1 · p. 68

6.2.3 Expert Scaling

• At Gemma 4 scale, expert delay is attractive because 8 selected experts contribute 23.7 MB per MoE layer, amortizing to 7.9 MB over two layers and making attention binding.Pre-dense routing also extends the expert-I/O window: loading the selected tiles takes approximately 0.47 ms per layer, roughly matching two attention layers at 50 GB/s RAM bandwidth.Source: §6.2.3 · p. 67

6.4 Limitations

• The evidence is limited to d = 512, L = 6 proof-of-concept training, single-token decode, and x86-64, while the 5.94 tok/s comparison is cross-architecture and Claim 6 is refuted whereas Claim 8 is inconclusive.cflow reaches 5.94 tok/s versus 4.75 for llama.cpp and 1.65 for vLLM, but differs in model and corpus; batched throughput, ARM NEON, and Apple Silicon are outside the report’s claims.Source: §6.4 · p. 69

7.1 Summary of Contributions

• cflow is a production-quality Rust streaming engine with tile-native formats, fused projections, conditional expert loading, AVX2+FMA kernels, and exact Rust↔PyTorch parity.Parity was confirmed on two trained architectures by 124 unit and integration tests with zero failures.Source: §7.1 · p. 70
• Five pipeline-native architectures rewrite transformer data flow for vertical pipelining; arch4 achieves 6.26 perplexity, while arch5 achieves the best parameter efficiency.arch2_4_combined combines δd = 1 and δe = 2 and is the only architecture with measured bandwidth reduction.Source: §7.1 · pp. 70–71

7.2 The Co-Design Thesis, Restated

• The co-design thesis is supported by bandwidth-bound arithmetic, 7.29× fewer L1-d read misses, and a 2.00× reduction in arch2_4_combined critical-path bandwidth.Five architectures trained to convergence, showing that dependency-graph rewriting can preserve useful learning.Source: §7.2 · p. 71

7.3.1 When the Result Matters

• The most relevant deployments are bandwidth-limited edge or consumer CPUs, large on-CPU MoE inference, and latency-critical single-token generation.Conditional expert loading can provide up to 16× E/k reduction at Gemma 4 scale.Source: §7.3.1 · p. 72

7.3.2 What Practitioners Should Take Away

• Conditional expert loading and L2-sized tile-streaming can be adopted independently, whereas dependency-graph co-design requires training-time architectural changes and cannot retrofit existing checkpoints.Conditional loading provides structural E/k bandwidth reduction; tile conversion requires a one-time checkpoint conversion.Source: §7.3.2 · p. 72

7.4 Future Work

• Future work prioritizes a controlled same-model comparison with llama.cpp and vLLM, followed by larger delays, attention-path delays, speculative recovery, new hardware backends, and batched decoding.The current comparison uses cflow’s 30.9B pipeline-native MoE but dense Qwen2.5-32B baselines, making same-model evaluation the most important remaining measurement.Source: §7.4 · pp. 72–73

7.5 Closing Remarks

• 5.94 tok/s end-to-end decode on a 30.9B pipeline-native MoE exceeded llama.cpp on the same CPU, while expert-delay overlap delivered a measured 1.68× net win.The closing results also report a 2.00× critical-path reduction and 7.29× fewer L1-d misses.Source: §7.4 · p. 72, §7.5 · pp. 73–74