PaperlayerView exact revision on arXiv

Source-linked AI summary

Pipeline-Native Transformers: Co-Designing Model Architecture and CPU Inference for Bandwidth-Efficient Autoregressive Decode

Tom Poperszky

arXiv:2608.23841v1cs.ARcs.LGcs.PF

TL;DR

Single-token CPU decoding is limited primarily by memory bandwidth, so the paper asks whether jointly designing the model and runtime can reduce per-token weight traffic. It introduces cflow with pipeline-native transformers and reports bandwidth, cache-locality, throughput, and failure-mode measurements, including a 2.00× critical-path bandwidth reduction for arch2_4_combined. The results support the co-design approach within the evaluated single-token setting, while leaving two claims unsupported and the end-to-end comparison uncontrolled across models.

  • Problem

    Single-token CPU decoding is bandwidth-bound, while GPU-derived runtimes retain layouts and execution assumptions that cause avoidable memory traffic, including loading unselected MoE experts.

  • Method

    The paper co-designs cflow, a tile-streaming CPU inference engine, with pipeline-native transformer architectures whose dependency graphs permit vertical stage-major execution.

  • Results

    2.00× critical-path bandwidth reduction is achieved by arch2_4_combined, supported by 7.29× fewer L1-d cache read misses; cflow also sustains 5.94 tok/s on a 30.9B pipeline-native MoE.

  • Takeaways & Limitations

    The measurements support co-design as a way to reduce CPU decode bandwidth, with dense delay controlling bandwidth and expert delay controlling quality among the trained candidates.

  • Takeaways & Limitations

    The evidence targets single-token decode, and the end-to-end throughput comparison is cross-architecture rather than a controlled runtime-versus-runtime comparison.

Abstract

from arXiv · show

Single-token autoregressive decode on CPUs is bound by memory bandwidth, not arithmetic: a modern CPU sustains roughly 1 TFLOP/s of compute but only about 50 GB/s from main memory, and each generated token must stream every active weight once. This report argues that the most effective response is to co-design the model architecture and the inference runtime together. It presents cflow, a CPU-first streaming engine, alongside a family of pipeline-native transformer architectures whose inter-layer dependency graphs are constructed to permit a vertical, stage-major execution schedule. cflow stores weights as L2-sized tiles in compute-consumption order, reads only the top-k experts of each mixture-of-experts layer, fuses projections, and executes a delay-aware schedule from per-model dependency parameters. Across five architectures trained on TinyStories, one (arch2_4_combined) achieves a 2.00x reduction in critical-path weight bandwidth (9.00 to 4.50 MB/token) within 0.24 perplexity of the best candidate, and the tile layout incurs 7.29x fewer L1-data read misses than a row-major baseline. On a 30.9-billion-parameter pipeline-native MoE, cflow decodes at 5.94 tokens/s (tok/s) on a 32-vCPU Ice Lake server, ahead of llama.cpp (4.75) and the vLLM CPU backend (1.65) on comparably sized dense models. Realizing the expert-delay window as asynchronous I/O overlap on a disk-resident expert tier yields a further net win of up to 1.68x, matching the overlap model within 1%. Measurement refutes one of the eight design claims and leaves a second inconclusive; both are reported in full, with the conditions under which they would hold.

Note on the Use of AI Tools

The report used Anthropic’s Claude to assist with implementing cflow and drafting and editing the report, while the author retained responsibility for research decisions, analysis, conclusions, and errors.

  • Claude assisted with implementing the cflow runtime and supporting code.
  • Claude also assisted with drafting and editing the report.
  • The author states that all research direction, architectural and experimental design, analysis, conclusions, and responsibility for errors remain their own.

Introduction

Single-token CPU decoding is governed primarily by memory bandwidth, motivating joint design of the model architecture and inference runtime. The report develops cflow and pipeline-native transformers, measures bandwidth and cache benefits, and identifies both successful and unsupported design claims within a defined scope.

  • The CPU bandwidth bottleneck: 1 TFLOP/s of CPU arithmetic and 50 GB/s of memory bandwidth leave single-token decode bandwidth-bound rather than compute-bound.Each generated token loads every active weight once, making byte traffic the principal latency determinant.
  • The GPU-first retrofit problem: 15-fold avoidable expert-weight overhead arises when a top-8 MoE runtime loads all 128 experts instead of only selected experts.The report identifies GPU-derived layout, execution-order, and expert-loading assumptions as the retrofit problem.
  • The co-design opportunity: Co-design makes per-token bandwidth reductions available by pairing cflow’s tile-streaming runtime with architectures whose dependency graphs permit vertical stage-major execution.cflow uses L2-sized tiles and consumption-order reads; pipeline-native architectures provide the relaxed dependencies needed by the schedule.
  • Measured contributions: 2.00× critical-path bandwidth reduction and 7.29× fewer L1-d cache read misses are measured for arch2_4_combined and tile-streaming, respectively.The bandwidth result is 9.00 MB/token to 4.50 MB/token at the trained geometry.
  • Negative results and scope: Six of eight claims hold; Claim 6 is refuted and Claim 8 is inconclusive because measured storage-to-RAM I/O leaves no overlap for the tested scheduler.The report gives conditions under which the unsupported claims could be revisited.
  • End-to-end throughput: cflow sustains 5.94 tok/s on a 30.9B pipeline-native MoE, ahead of llama.cpp at 4.75 tok/s and vLLM at 1.65 tok/s on the same CPU.The comparison is separate from the eight structural claims and uses parameter-comparable dense baselines.
  • Architecture trade-offs: arch2_4_combined achieves 2.00× bandwidth reduction, while arch4_async_experts achieves the best perplexity at 6.26 versus 6.50.Dense delay acts as the bandwidth knob, whereas expert delay acts as the quality knob; the two architectures occupy opposite trade-off corners.
  • Negative results and scope: The evidence targets single-token autoregressive decode and does not claim generalization to batched inference.Batched decoding shifts arithmetic intensity toward compute-bound operation and may reduce the relative benefit of bandwidth optimization.

Background

Single-token transformer decoding on CPUs is dominated by memory bandwidth because Q4 matrix-vector operations have arithmetic intensity well below machine balance. The report therefore targets weight traffic through cache-aware tiling, selective expert loading, and model-runtime co-design.

  • Cache-Aware Execution: 8–10× separates L1 and main-memory bandwidth, motivating tiles that keep activations cache-resident while streaming weights.The tile-streaming design exploits this gap at L2 granularity.
  • CPU Memory Bottleneck: 40–50 GB/s of sustained main-memory bandwidth determines single-token inference latency on representative CPUs.The report contrasts this with much higher cache bandwidth and emphasizes the memory-hierarchy gap.
  • CPU Memory Bottleneck: 4 FLOP/byte for Q4 matrix-vector products is five times below the 20 FLOP/byte machine balance.Thus single-token inference is memory-bandwidth-bound rather than arithmetic-bound.
  • MoE Bandwidth: MoE routing can avoid substantial traffic by loading only selected experts instead of all E experts.For Gemma 4’s E = 128 and k = 8 geometry, selective loading reduces approximately 379 MB of all-expert data to 23.7 MB per layer at Q4.
  • Model-Runtime Co-Design: Pipeline-native architectures rewrite inter-layer dependencies so vertical stage-major execution becomes mathematically valid.This extends prior parallel attention–FFN ideas with dense and expert delays co-designed with the runtime schedule.

The cflow Runtime

cflow is a CPU-first streaming runtime organized around cache-sized tiles, compute-order storage, fused kernels, and conditional expert fetching. Its formats and headers encode the model-specific schedule, while measurements show both effective prefetching and a negative result for explicit cache hints.

  • Runtime Components: cflow combines L2-sized tile storage, .cflow and .vflow formats, fused CPU computation, and conditional expert prefetching.The runtime is built specifically around CPU memory hierarchies and single-token transformer inference.
  • Tile Design: Compute-order tile layout makes sequential reads follow forward-pass consumption and separates data-dependent expert tiles into a random-access bank.This supports OS readahead for ordinary weights and selective reads after routing.
  • Tile Design: 18 KB tiles fit multiple quantized weight chunks in 256 KB L2 while keeping the activation vector in L1.A 128×256 Q4 tile contains 16 KB of quantized data and 18,448 bytes including metadata.
  • Vertical Streaming: The .vflow format groups tiles by execution stage across layers, enabling sequential attention and dense-FFN reads for vertical pipeline groups.The intended sequentiality is measured rather than assumed to be exploited by hardware or the operating system.
  • Measured Runtime Behavior: No measurable benefit appears from explicit _mm_prefetch when storage-to-RAM bandwidth is the bottleneck.PF=1 and PF=0 remain within measurement noise at both reported model scales, and the claim remains untestable when compute dominates I/O.

Pipeline-Native Transformer Architectures

The report addresses the remaining bandwidth bottleneck by co-designing transformer dependency graphs with a vertical pipeline schedule. This architectural change complements cflow’s cache-optimal streaming infrastructure.

  • Pipeline-Native Design: Pipeline-native architectures rewrite inter-layer dependency graphs so cflow’s vertical pipeline schedule is valid by construction.The architecture is designed around the execution schedule rather than merely adapted to a target device.

4.1 The Layer Dependency Problem

Standard transformer dependencies force sequential layer execution, preventing stage-major weight streaming. The report frames the opportunity as reducing critical-path bandwidth rather than total weight bytes.

  • Sequential Dependency: Layer ℓ+1 cannot begin until layer ℓ produces both attention and FFN outputs in a standard pre-norm transformer.The next layer’s normalized input depends on the preceding layer’s complete residual output.
  • Sequential Dependency: A standard layer stack therefore requires each layer’s weights to be fully read before the next layer begins.This dependency prevents reordering the weight-read schedule without changing the architecture.
  • Vertical Schedule: A vertical schedule overlaps attention and FFN stages from adjacent layers instead of completing layers serially.Its validity requires the next layer’s attention not to depend on the previous layer’s FFN output.
  • Vertical Schedule: The vertical schedule is mathematically invalid for the standard architecture because the next layer includes the preceding FFN contribution.This motivates dependency-relaxation architectures.
  • Critical-Path Bandwidth: 9.00 MB/token is the undelayed critical-path bandwidth, while arch2_4_combined reaches 4.50 MB/token through delayed dense and expert reads.The 16.5 MB fully serial total is not the latency-setting quantity because concurrent streams make the per-layer path the relevant measure.

4.2 DAG Rewriting: Dense Delay and Expert Delay

The architectures rewrite dense and expert dependencies with configurable delays, enabling stage-major execution while preserving distinct forward-pass semantics.

  • Dense delay: Dense delay lets a layer’s FFN consume an earlier residual, breaking its dependency on the current layer’s attention output.The delay is recorded as dense_delay and used by the scheduler to select residual snapshots from a ring buffer.
  • Expert delay: Expert delay routes at one layer but injects expert outputs δe layers later, creating a window for overlapping expert-tile I/O with intervening computation.The expert outputs are queued after routing and dequeued at their target layer; expert_delay stores this choice in the model header.
  • Combine styles: ParallelSqrt2 runs dense and MoE branches in parallel on the same normalized residual, sums and scales their outputs, and uses no delays.This convention corresponds to δd = δe = 0.
  • Combine styles: DelayedSum combines a one-layer dense delay with a two-layer expert delay, routing from the post-attention normalized residual and summing delayed outputs.It is used by arch2_4_combined with δd = 1 and δe = 2.
  • Combine styles: AsyncExperts keeps δd = 0, uses δe = 2, and routes from the pre-attention normalized residual before injecting expert outputs later.This semantics is used by arch4_async_experts.

4.3 The Five Candidate Architectures

Five architectures were trained under a shared TinyStories configuration and tested different dependency, expert-delay, and weight-sharing strategies.

  • Shared setup: Five candidates share TinyStories training, 10K steps, AdamW, a 50K GPT-2 vocabulary, and nominal geometry d = 512, L = 6, and dff = 2048.Architectures 4 and 5 add MoE and weight-sharing components that increase parameter counts.
  • Arch1: Arch1 decouples attention and FFN residual streams, synchronizing them every three layers, but achieves 1.00× bandwidth reduction and test perplexity 7.21.With δd = 0 and δe = 0, its pipeline opportunity does not reduce critical-path bandwidth.
  • Arch2_4_combined: Arch2_4_combined uses δd = 1 and δe = 2 to break attention–FFN dependencies and overlap expert-tile loading with intervening computation.Its critical-path bandwidth is 4.50 MB/token versus 9.00 MB/token undelayed, a 2.00× reduction, with test perplexity 6.50.
  • Arch2_4 ablation: Removing the expert queue leaves arch2_4’s bandwidth at 4.50 MB/token and changes test perplexity from 6.50 to 6.52, within run-to-run noise.The ablation gives up the expert-side overlap window later used for I/O hiding.
  • Arch3: Arch3 exposes attention and transformed outputs as pipeline registers, but with no delays achieves 1.00× bandwidth reduction and test perplexity 7.24.It serves as a delayed-residual baseline without additional staleness beyond the specified register dependencies.
  • Arch4: Arch4 applies only expert delay with pre-dense routing, reaching the best test perplexity 6.26 but retaining undelayed critical-path bandwidth.Its perplexity is 0.24 better than arch2_4_combined’s 6.50.
  • Arch5: Arch5 shares two weight blocks across six effective layers, yielding 14.06 MB/token critical-path bandwidth, test perplexity 6.77, and approximately 4.69 MB of unique on-disk weights.The architecture trades bandwidth for parameter efficiency through temporal weight reuse.

4.4 The Delay-Aware Scheduler

The delay-aware scheduler uses model-provided delay parameters to manage residual history and expert queues while executing layers in order.

  • State management: The scheduler maintains a residual ring buffer of max(δd, δe) + 1 entries and uses header-provided delays without model-family-specific logic.Each layer selects the appropriate delayed residual by index modulo the buffer size.
  • Layer execution: For each layer, the driver retrieves the delayed residual, invokes the executor with current and delayed state, handles expert outputs, and updates the ring buffer.AsyncExperts dequeues scheduled outputs before execution, while DelayedSum handles its queue internally.
  • Memory overhead: At d = 512 and δe = 2, residual and pending-expert state add approximately 10 KB of working memory.The overhead comprises a residual ring buffer plus at most two pending expert vectors.

4.5 Bandwidth Model

The bandwidth model treats critical-path traffic as the longest concurrently issued stream and shows dense and expert delays as complementary bandwidth controls.

  • Critical-path definition: The critical path is the weight-read sequence that must finish before the current token’s output logit is available.Delayed schedules move some reads off this path.
  • Delay effects: A dense delay of δd spreads dense FFN bytes across δd + 1 layers, reducing their per-layer critical-path contribution to Bdense/(δd + 1).The stream leaves the critical path once its amortized contribution falls below the attention stream.
  • Delay effects: An expert delay of δe spreads selected expert-tile bytes across δe + 1 layers, reducing their per-layer contribution to Bexpert/(δe + 1).This applies to the top-k tiles selected at the routing layer.
  • Baseline: With no delays, the longest 1.50 MB layer stream across six layers gives Bnaive = 9.00 MB per token.The streams are issued concurrently, so the critical path uses their maximum rather than their sum.
  • Arch2_4_combined: With δd = 1 and δe = 2, arch2_4_combined reduces the longest per-layer stream to 0.75 MB and Bcritical to 4.50 MB per token.This is a 2.00× reduction relative to the 9.00 MB/token naive critical-path baseline.
  • Interpretation: The fully serial 16.5 MB total is not the same quantity as the 9.00 MB critical-path baseline, which already credits intra-layer overlap and is the correct comparison.The figures are analytic outputs from geometry and delay integers, not runtime timing measurements.
  • Complementary knobs: At the arch2_4 geometry, dense delay has twice the per-layer bandwidth impact of expert delay, while at Gemma 4 scale expert delay would dominate.Arch4 separately demonstrates a quality improvement of 0.24 over arch2_4 at the cost of sacrificing dense bandwidth reduction.
  • Evaluation summary: Table 4.1 summarizes the five architectures’ training results and bandwidth analysis, while Figure 4.1 plots their bandwidth–quality trade-off.The table covers models trained 10K steps on TinyStories with d = 512 and L = 6.

4.6 The Co-Design Trade-off

The pipeline-native experiments reveal a tunable trade-off among bandwidth, quality, parameter count, and implementation complexity. Arch2_4_combined delivers the strongest bandwidth reduction, while other designs favor quality or storage efficiency.

  • Bandwidth vs. quality: 2.00× bandwidth reduction makes arch2_4_combined the strongest bandwidth design, while arch4 achieves the best quality at ppl 6.26.Arch2_4_combined reaches ppl 6.50; arch4 gives up the dense bandwidth reduction.
  • Bandwidth vs. quality: No single architecture is Pareto-optimal; bandwidth and quality remain tunable through (δd, δe) and the routing anchor.Arch2_4_combined favors bandwidth, whereas arch4 favors a cleaner routing signal and quality.
  • Quality vs. parameter count: Competitive quality at one-third the parameter count makes arch5 preferable when model storage, rather than per-token bandwidth, is the constraint.Arch5 reaches ppl 6.77, with 4.69 MB of unique on-disk bytes despite 14.06 MB of counted bandwidth.
  • Complexity vs. generality: Modest implementation costs support the trade-off: the scheduler uses a ring buffer and expert queue with O(δ · d) memory and header-parameterized dispatch.The runtime adds a conditional branch per layer but avoids model-family hard-codes.
  • Scaling behavior: 3× larger dense streams preserve the full effect of a one-layer dense delay in arch2_4 variants, provided delayed streams remain on the critical path.The 31B measurement confirms this geometry-dependent behavior, tying the benefit to streams that remain binding.

Evaluation

Evaluation shows that pipeline-native designs can train effectively while delivering substantial cache, bandwidth, and decode-throughput benefits, though two design claims fail or remain unproven under tested conditions.

  • Training and quality: All five architectures converge on TinyStories, with perplexity ranging from 6.26 to 7.24 despite small model scale.Arch1 and arch3 score 7.21 and 7.24, while arch4 reaches 6.26.
  • Expert loading: Top-k expert loading reduces expert reads structurally: E = 8, k = 2 reads 25% of expert bytes, while E = 128, k = 8 yields a 16× reduction.The offset-table design selects and reads only the routed experts’ tiles.
  • Cache locality: 7.29× fewer L1-d read misses are measured for tiled dense-down projection than the row-major baseline at the trained 8.34B geometry.The measured counts are 2.64M tiled versus 19.3M naive misses.
  • Negative results: Explicit prefetch provides no measurable storage-bandwidth benefit and can harm decode, with disabling it raising throughput from 2.58 to 4.96 tok/s.The prefetch thread consumed approximately 48% of decode time walking expert regions and added cache pressure.
  • Asynchronous overlap: 1.68× is the maximum measured net win from asynchronous expert-I/O overlap, while the overlap model matches all four measured points within 1%.The staged schedule reaches the disk floor instead of paying serial compute plus I/O.
  • Negative results: Claim 8 remains inconclusive because stage-major layout benefits require asynchronous overlapped I/O, while the current scheduler does not overlap I/O and compute.Claim 6 is refuted under storage-bound conditions where PREFETCHT0 cannot overlap the limiting transfer.

Discussion

The discussion frames pipeline-native co-design as a bandwidth optimization whose benefits depend on the binding weight stream, model scale, hardware, and deployment setting. It also identifies unresolved validation boundaries, including quality-matched comparisons, larger-scale training, and attention-path delays.

  • Scaling and binding streams: 2.00× critical-path bandwidth reduction for arch2_4_combined is analytically derived from its geometry and delay parameters, not favorable hardware conditions.At the trained geometry, dense-stream delay changes the maximum from 1.50 to 0.75.
  • Scope and limitations: The 2.00× result is tied to proof-of-concept geometry, so larger-scale bandwidth reduction requires a separate calculation.The relevant ratio of delayed bytes to total critical-path bytes changes with model dimensions and expert contribution.
  • Scope and limitations: The PMU measurement reports 7.29× fewer L1-d misses at 8.34B parameters on Sandy Bridge, while a Ryzen wall-clock proxy at the same geometry shows 1.12–1.36× speedup.These are complementary measurements across different instruction sets, not a single hardware result.
  • Scaling and binding streams: The co-design pays off when a delayable dense or expert stream sets the per-layer critical path; attention becomes the floor after amortization.At larger scales, attention-path delay is proposed, but KV-cache staleness makes it a future-work problem.
  • Scaling and binding streams: At Gemma 4 scale, expert bandwidth is the binding MoE constraint before delay, while δe = 2 lowers it below attention and exhausts the expert-delay benefit.The expert stream is 23.7 MB/layer versus 15.8 MB for attention, then amortizes to 7.9 MB.
  • Scope and limitations: The end-to-end cflow comparison is not quality-controlled because it uses a pipeline-native MoE against dense baselines with different architectures and training corpora.A same-model comparison is identified as the most important remaining measurement.
  • Scope and limitations: Explicit prefetch is ineffective at tested scales when I/O binds, whereas asynchronous expert overlap can help on disk-resident tiers; Claim 6 is refuted and Claim 8 inconclusive.The prefetch mechanism could benefit systems with sufficient RAM or sufficiently fast NVMe, while larger-scale training may expose delayed-gradient instabilities.

Conclusion

The report contributes a CPU-first streaming runtime and five transformer architectures designed for vertical stage-major execution. Its evidence combines convergence, cache locality, analytical bandwidth reduction, end-to-end throughput, and explicit reporting of comparison and deployment limits.

  • Contributions: cflow is a Rust CPU-first streaming engine co-designed with pipeline-native transformers whose dependency graphs permit vertical stage-major pipelining.The runtime includes tile-native formats, fused projections, conditional expert loading, AVX2 kernels, and staged direct-I/O expert fetch.
  • Contributions: Five architectures span residual-stream, delayed-dense-and-expert, pipeline-register, asynchronous-expert, and fixed-point weight-sharing designs.arch2_4_combined is the bandwidth-oriented variant, arch4 reaches 6.26 perplexity, and arch5 targets parameter efficiency.
  • Validation: 7.29× fewer L1-d read misses were measured at the 8.34B geometry, while arch2_4_combined analytically reduces bandwidth from 9.00 to 4.50 MB/token.The bandwidth result is also realized with up to 1.68× net wall-clock improvement on a disk-resident expert tier.
  • Thesis: The bottleneck for single-token CPU inference is memory bandwidth, motivating joint redesign of model dependencies, tile layout, file format, and execution schedule.Single-token matrix-vector arithmetic intensity is 4 FLOP/byte versus a 20 FLOP/byte machine balance.
  • Validation: Five trained architectures demonstrate that dependency-graph rewriting can preserve useful learning while enabling pipeline-oriented execution.The conclusion reports real-data training to convergence alongside cache and bandwidth validation.
  • Practical adoption: Conditional expert loading and tile-streaming can be adopted independently, whereas dependency-graph co-design requires training-time architectural changes and is not retrofittable.The latter is the most powerful but highest-friction intervention.
  • End-to-end evaluation: 5.94 tok/s on a 30.9B pipeline-native MoE exceeded llama.cpp at 4.75 tok/s and vLLM at 1.65 tok/s on the same Ice Lake CPU.The comparison remains cross-architecture and therefore is not quality-matched.
  • Closing assessment: The report concludes that two design claims failed or remained inconclusive, clarifying where the co-design assumptions stop holding.It highlights explicit prefetch failure under I/O binding and absent stage-major readahead benefit at tested scales.
Loading 2608.23841v1…