PaperlayerView exact revision on arXiv

Source-linked AI summary

Daedalus-150M: A Convolution-Attention Hybrid Designed for CPU Inference

Christos Koutsiaris

arXiv:2608.20210v1cs.IRcs.AIcs.CLcs.LG

TL;DR

Small language models often inherit architectures designed for GPU serving, despite batch-size-one CPU decoding being constrained by memory traffic and growing attention caches. Daedalus-150M instead combines mostly fixed-state short convolutions with a minority of attention layers and evaluates that choice against pre-registered criteria. The hybrid reaches 47.31 against a 42.20 bar and outperforms its matched all-attention twin on the chosen quality metric while decoding faster at long context.

  • Problem

    Batch-size-one CPU decoding is constrained by weight streaming and attention caches whose per-token cost grows with context length.

  • Method

    The paper designs an 18-block hybrid with twelve short-convolution blocks and six attention blocks, then tests it using pre-registered, parameter-matched comparisons.

  • Results

    47.31 is the full-scale five-task score against a 42.20 pre-training bar, while the matched hybrid wins the chosen quality metric by 0.81% and decodes 1.76× faster at 2048 tokens.

  • Takeaways & Limitations

    The results support fixed-state convolutions plus limited attention as a viable CPU-oriented architecture choice for small language models.

  • Takeaways & Limitations

    The 2 T-token peer remains ahead by 3.9 points, reflecting a stated quality–decode-speed trade at fixed model size.

Abstract

from arXiv · show

Small language models are usually built like large ones and then squeezed onto a CPU afterwards. We did the opposite: we fixed the target first, one user, one token at a time, 4-bit weights, ordinary CPU, and chose the architecture to suit it. The result keeps full attention in only 6 of its 18 blocks. The other 12 use short convolutions whose memory is two timesteps wide no matter how long the conversation gets, so two thirds of the network never re-reads a growing cache. Trained from scratch on 59.9B tokens, the model scores 47.31 on a five-task benchmark against a bar of 42.20 that was fixed before training began. It beats GPT-2 124M, Pythia-160M, OPT-125M and GPT-neo-125M, all trained on three to six times more data, and exceeds MobileLLM-125M's published score despite that model seeing a trillion tokens. Validation bits-per-byte is 0.8685. To check the architecture rather than the training recipe, we trained a conventional all-attention model of the same size on the same data, and wrote down the winning condition before scoring either. The hybrid won the chosen quality metric by 0.81%, matched it on downstream tasks, produced a 6.3% smaller 4-bit file, and decoded 1.76x faster at 2048 tokens of context, 2.08x against an external model of similar size. In every measurement the speed advantage is near zero at an empty context and grows with length, which is what the mechanism predicts and what a merely leaner model would not show. A simple bandwidth calculation predicts only 1.17x, so memory volume alone does not explain the gap. We also report what did not work: an unmitigated 4-bit quality cost, roughly half the convolution channels ending up inert and impossible to remove, and a vocabulary larger than this model size warrants.

I. INTRODUCTION

Daedalus-150M starts from batch-size-one CPU constraints, where memory traffic and growing attention caches dominate decoding, and chooses a hybrid architecture to address them. The paper evaluates this choice with pre-registered criteria, matched comparisons, and context-dependent speed measurements.

  • Batch-size-one CPU decoding is governed by weight bytes read per token because there is no batching to amortise weight loading.
  • Attention cache reads grow linearly with context length and can dominate long-context decoding on CPUs.This motivates replacing most attention layers with constant-state mechanisms.
  • The evaluation fixes the quality bar and uses parameter-matched ablations plus internal and external decode comparisons across context depths.The criteria were fixed before headline results, and the peers were measured on a common harness.
  • The architecture uses mostly fixed-state short convolutions while retaining minority attention layers to balance CPU efficiency with retrieval capability.The design follows a memory-traffic argument specific to batch-size-one CPU decoding rather than adapting GPU practice.
  • Grouped-query attention and 4-bit quantisation complement the hybrid design by reducing cache or deployment cost for the retained attention layers.The paper treats these choices as complementary to, rather than substitutes for, the cache-free layer composition.

III. ARCHITECTURE

Daedalus-150M has an 18-block stack combining six attention blocks with twelve short-convolution blocks. The convolution blocks use depthwise length-3 filters and maintain a two-timestep recurrent state, making their decode cost independent of context length.

  • Daedalus-150M contains 160.49 M parameters across 18 blocks, with six attention blocks and twelve short-convolution blocks.The attention blocks occur at indices 4, 7, 9, 11, 13, and 16.
  • The block sequence interleaves convolution and attention layers rather than clustering the attention layers.
  • Each convolution block uses a depthwise length-3 convolution with groups equal to the channel count.Channels evolve independently in the convolution operation.
  • The convolution state is exactly two timesteps wide regardless of context length.The gating terms B and C provide input-dependent behaviour beyond a fixed kernel.
  • Convolution decoding costs the same at token 2000 as at token 2, unlike attention decoding.

B. Why six attention layers

Six attention layers are retained to preserve long-range retrieval while twelve fixed-state convolution layers reduce cache traffic and context-dependent costs. A first-order byte model predicts the direction of the speed gain but substantially underestimates its magnitude.

  • Why six attention layers: Six distributed attention blocks preserve precise long-range retrieval while avoiding an all-convolution architecture.The retained attention layers are spread through the depth to distribute retrieval capacity across representation levels.
  • Why six attention layers: The hybrid maintains a key–value cache in six layers rather than twenty-four in the all-attention twin.
  • Why six attention layers: Grouped-query attention uses four key–value heads for twelve query heads, further reducing cache size in the retained attention layers.This reduces the cache by a factor of three on those layers.
  • Why six attention layers: For the hybrid, κhyb = 6144 B, while the dense twin has κdense = 12,288 B per context token.The dense twin’s cache is twice as large because it has four times as many attention layers despite narrower per-layer cache.
  • Why six attention layers: 17 % is the predicted advantage at depth 2048, compared with a measured 76 % advantage.The discrepancy is attributed to latency-bound dependent softmax reductions, cache locality, and the twin’s additional layers.
  • Why six attention layers: Attention-kernel improvements would narrow but not close the gap because the hybrid avoids keeping much of the cache.

V. CORPUS AND DATA PIPELINE

Training uses a ten-source English mixture of 16.93 B unique tokens and a 59.9 B-token budget. Source repetition is capped at four epochs, with freed mass redistributed across sources that retain headroom.

  • The training corpus contains 16.93 B unique tokens from ten English sources weighted toward educational and reasoning-dense text.The evaluation suite is knowledge- and commonsense-heavy.
  • A 59.9 B-token budget implies roughly 3.5 epochs over the unique-token corpus.
  • Each source is capped at four epochs rather than repeated arbitrarily to meet its nominal share.Mass freed by capped sources is redistributed by water-filling across sources with remaining headroom.
  • The cap prevents tiny sources from being repeated thousands of times to fill a nominal mixture share.

B. Packing and held-out data

The training pipeline uses whole-shard holdouts and a warmup–stable–decay schedule, then tests the architecture with a pre-registered, parameter-matched comparison. Validation bits-per-byte separates the models more clearly than the downstream suite at this training scale.

  • Packing and held-out data: ≈2% holdouts reserve whole shard files per source, preventing training windows from crossing train/holdout boundaries.Because shards are indivisible, realised holdout fractions vary by source; one source reaches 9.16%.
  • Training schedule: The schedule holds the peak learning rate from approximately step 20,000 to step 68,461, so loss plateaus before decay.The plateau is expected rather than a training stall; most remaining quality arrives during the decay phase.
  • Controlled ablation: 160.49 M versus 161.25 M parameters: the hybrid and dense twin were trained on identical data and schedules under a pre-registered 0.5% validation margin floor.Both arms were fully decayed after 5 B tokens, with validation bits-per-byte computed over 645 M held-out tokens.
  • Ablation quality: 0.81%: the hybrid wins the pre-registered validation metric, while the dense twin is nominally ahead by 0.14 points on the five-task mean.The per-task results split in both directions, with the hybrid leading HellaSwag and OpenBookQA and the twin leading WinoGrande, PIQA and ARC-Easy.
  • Evaluation interpretation: Validation bits-per-byte separates the architectures more sensitively than downstream tasks at the 5 B-token budget, where roughly two of five tasks sit near chance.WinoGrande scores 50.0 and 51.6 against a 50.0 chance floor, so the downstream comparison is treated as a quality tie.

B. Per-domain difficulty

Validation difficulty varies substantially by source, while CPU decode measurements show the hybrid’s advantage increasing with context. The observed speed gap exceeds a bandwidth-only prediction and is tied to cache traversal, locality, and layer-count differences.

  • Per-domain difficulty: Nearly a factor of two: per-source validation bits-per-byte varies widely, with code and encyclopedic text more predictable than broad filtered web text.Aggregate validation therefore depends strongly on mixture weights as well as model quality.
  • CPU decode speed: 1.76× at trained context length: the hybrid’s decode advantage grows monotonically with context, while its empty-context advantage is negligible.The measurement uses 4-bit weights, 8 threads, 128 generated tokens, and alternating arms within a single pass.
  • Measurement caveats: 1.15–1.17× is the depth-zero floor, not the headline result, and 1.29× came from a non-alternating measurement that does not reproduce.These historical figures should not be quoted as the architecture’s demonstrated speed advantage.
  • Interpretation: The curves diverge rather than differ by a constant, making context-dependent separation—not absolute throughput—the architectural comparison.The dense twin rereads 12,288 B of cache per context token versus 6,144 B for the hybrid.

D. Comparison against an external peer

Against an external peer, Daedalus preserves its context-dependent decoding advantage despite carrying more parameters, while its quantised artefact is smaller. The broader evaluations also show strong benchmark performance, though a 2 T-token peer remains ahead.

  • External peer: 2.08× faster decoding at the target context was achieved against an external peer while carrying 19% more parameters.The same near-unity-at-zero-context, growing-with-context signature also reproduced externally.
  • Quantised artefact size: 6.3% smaller 4-bit file resulted for the hybrid than for the dense twin, despite half-precision parameter-matched sizes within 0.5%.The result depends on tensor shapes rather than weight values, so it was fixed before training completed.
  • Benchmark comparison: 44.7 on the five-task harness was reached by the 5 B-token hybrid ablation, clearing the 42.2 bar and exceeding four 300 B-token-class peers on every task.This is the ablation result, not the finished Daedalus result.
  • Benchmark comparison: 47.31 on the five-task benchmark cleared the 42.20 bar by 5.11 points and beat every designated comparison model.The model also exceeded MobileLLM-125M’s published figure, which used a trillion tokens.
  • Benchmark comparison: 3.9 points separated the finished model from the 2 T-token peer that remained ahead.The paper explicitly frames this as a quality-versus-decode-speed trade-off at fixed size.
  • Quality and quantisation: 0.8685 validation bits-per-byte was achieved over a 645 M-token held-out set, while 4-bit quantisation incurred roughly a 6% perplexity cost.The quantisation comparison was 9.18 at half precision versus 9.75 at Q4_0 on held-out encyclopedic text.

IX. LIMITATIONS

The training run deviated from its intended mixture and continuity assumptions. These deviations are quantified, disclosed, and bounded rather than hidden.

  • A. The training mixture drifted from its target: 10.42 L1 percentage points measured mixture drift, exceeding the pre-fixed limit of 10.0.The largest single-source deviation was −0.24 points, described as negligible against benchmark noise.
  • A. The training mixture drifted from its target: Four repetitions capped each source, causing freed allocation to flow toward smaller sources with unread text.The realised mixture therefore differed from the intended one.
  • B. Checkpoint resumption: Training was interrupted and resumed from a checkpoint, leaving permanent properties in the released weights.The paper reports these consequences explicitly rather than smoothing them over.
  • B. Checkpoint resumption: Momentum restarted from zero because optimiser state was not carried across, then re-warmed over a few hundred steps.The paper characterises this as mild because the learning rate had already decayed to 0.0035.
  • B. The last 8 % of training used a slightly smaller corpus: 4.8 B final tokens formed a fresh corpus pass after the data cursor reset, so some documents were repeated and others omitted.Those tokens came from a snapshot 0.42 B tokens smaller than the earlier corpus.

C. Quantisation-aware training did not run

Quantisation-aware training was disabled after its first step produced a non-finite loss, leaving the released model with the full 4-bit quality penalty. The section also identifies substantial inefficiency from dead convolution channels and an oversized vocabulary.

  • Quantisation-aware training: A non-finite loss on the first step disabled the planned final 5% of exact-grid fake quantisation training.The quantisation grid was verified against the deployment runtime’s kernel.
  • Quantisation-aware training: About 6% perplexity, rather than 2.5% at smaller scale, is the released model’s 4-bit penalty after post-training quantisation.Half-precision weights remain available, so the quantisation-aware pass or another format could be applied without retraining.
  • Vocabulary: At 49,152 tokens, the vocabulary embedding table holds 37.7 M parameters, or 23% of the model, whereas 32 k would hold 24.6 M.The authors identify a smaller vocabulary as the first successor change.
  • Dead channels: Approximately 47.9% of short-convolution channels are inert, representing roughly 13.6 M parameters and an 8.5% parameter inefficiency.The fraction remained stable across the reported training steps.
  • Dead channels: Export-time pruning failed because the reference runtime requires short-convolution tensors at fixed model width and rejects narrowed artifacts.Reclaiming the parameters would require patching the runtime and forfeiting stock-binary compatibility.

XI. EVALUATION METHODOLOGY

The evaluation uses pre-registered, same-harness comparisons and separates validation quality from decode throughput. Measurements target the single-user CPU regime, where context depth and memory traffic determine the hybrid’s practical advantage.

  • Quality evaluation: Published peer scores run 0.5–1.5 points above locally re-scored checkpoints because task subsets and normalisation differ.The paper therefore compares peers on the same harness rather than against published figures.
  • Quality evaluation: Bits per byte is computed on a 645 M-token holdout to make validation quality comparable across tokenisers.Per-token metrics could favor a model with a more efficient tokeniser at equal predictive quality.
  • Decode evaluation: Decode throughput generates 128 tokens after priming the stated context depth, using 8 threads, 4-bit weights, repeated measurements, and alternating arms.The alternating protocol avoids background-load drift observed in an earlier back-to-back measurement.
  • Decode evaluation: Prompt processing is excluded because it is compute-bound and parallel over positions, unlike the memory-bound generation regime being tested.Including prompt processing would flatter the hybrid on workloads that decode little.
  • Deployment setting: Q4_0 prioritizes target-hardware kernel speed over lower-error formats, with a measured 2.5% perplexity cost at 5 B tokens.Formats such as Q4_K would reduce error while giving back part of the speed.
  • Deployment setting: At 2048-token context, the 95.56 MiB artifact plus an approximately 12.6 MB cache fits under 128 MB for a single-user session.The model can remain resident alongside an application rather than requiring a dedicated process boundary.
  • Deployment setting: The hybrid saturates memory bandwidth at a lower thread count than the dense twin because it moves fewer bytes per token.All reported thread-scaling figures use 8 threads.

XIII. DISCUSSION

The discussion frames the hybrid’s benefit as context-dependent: it is small for empty contexts and grows for long prompts, while several engineering and scope limits remain. The authors also identify retrieval evaluation and multi-seed replication as open work.

  • Discussion: Deployments with long conversations, retrieved documents, or files are expected to benefit most, while short independent prompts see little advantage.The distinction follows the reported context-depth dependence of decoding speed.
  • Discussion: The measured ratio is not reported beyond 2048 tokens because the model was trained only to that context length.The authors note that the equation implies further growth but decline to claim a usable extrapolation.
  • Discussion: Reducing attention below six layers may trade against retrieval capability, which was not measured and requires retrieval-sensitive evaluation.The remaining cache is concentrated in the six retained attention layers.
  • Open work: Planned follow-up includes diagnosing quantisation-aware training failure, addressing dead channels at initialization, testing a depth ablation, and replicating the ablation across seeds.A retrieval-sensitive evaluation is also listed as future work.
  • Conclusion: At full scale, the model scored 47.31 against a pre-training bar of 42.20 and achieved 0.8685 validation bits-per-byte.It beat every cited size-class model despite those models using three to six times more data.
  • Conclusion: The remaining issues are an unmitigated 4-bit penalty, inert convolution channels, and an inherited vocabulary larger than this model size warrants.The authors characterize all three as cheaper to fix in the next model than to retrofit.
Loading 2608.20210v1…