Source-linked AI summary
DeepInfer-Mac: Running a 750B-Class MoE on a 24 GB Laptop
Haorui Li
TL;DR
A 750B-class model exceeds a 24 GB laptop’s memory, and serial SSD reads leave each generated token waiting on storage. DeepInfer-Mac keeps frequent weights resident and loads routed experts into reusable GPU buffers, coordinating cache allocation with buffer lifetimes. It generates text locally, and matched decode comparisons report a 21.0% throughput gain from next-layer speculation over resident-expert overlap, alongside additional reads.
Problem
A 750B-class checkpoint occupies 338 GiB, far beyond a 24 GB laptop, while serial router-dependent SSD reads constrain token generation.
Source: §1 · p. 1Method
DeepInfer-Mac keeps frequent weights resident and loads selected experts from SSD directly into reusable GPU buffers, coordinating cache allocation with GPU buffer lifetimes.
Source: §1 · p. 1, §3.2 · p. 3Results
21.0% higher aggregate decode throughput comes from next-layer speculation over resident-expert overlap, with 5.4% more expert data read.
Source: §1 · p. 2Takeaways & Limitations
A laptop can execute a model whose stored parameters far exceed its memory when the active working set is small enough and remaining weights can be supplied from storage.
Source: §8 · p. 8Evidence covers one laptop, one converted checkpoint and a single sequence at a time, and the generated examples are not a broad quality benchmark.
Source: §8 · p. 8
Abstract
from arXiv · showhide
DeepInfer-Mac runs the 750B-class DeepSeek-V4.1-Flash model on a MacBook Air with 24 GB of unified memory. A mixed low-bit conversion occupies 338 GiB on the internal SSD. About 7.6 GiB of weights remain resident; routed experts and Engram rows are loaded when needed. This separation makes the model executable, but it leaves each generated token waiting for a succession of storage reads. We organize expert weights so that reads fill reusable GPU buffers directly, and couple cache allocation with the lifetime of those buffers. In end-to-end runs, the system generates English, Chinese and code at 0.21-0.72 tokens per second, with a largest sampled process footprint of 16.73 GiB across the tested requests. Matched decode comparisons show a 21.0% throughput gain from next-layer prefetching over resident-expert overlap alone, at the cost of additional reads. Longer prompts remain costly. We describe the design, provide the implementation and measurement scripts, and identify the limits of this operating point: a model far larger than memory can run locally, while storage and prefill still govern the user's wait.
1 INTRODUCTION
DeepInfer-Mac runs a 750B-class MoE on a 24 GB laptop by keeping frequently used weights resident and loading routed experts and Engram tables from SSD. End-to-end runs generate text, while matched decode tests find a throughput gain from next-layer speculation at the cost of additional reads.
1 INTRODUCTION
- 338 GiB of compressed checkpoint weights fit on SSD while frequently used weights remain resident and selected experts move through memory.DeepSeek-V4.1-Flash selects six of 384 routed experts in each of its 40 layers.Source: §1 · p. 1
- A one-token-at-a-time laptop execution path makes SSD bandwidth and the router-to-read dependency chain govern generation pace.The router must run before the system knows which experts to read, and those reads must finish before the missing experts can be computed.Source: §1 · p. 1
- The design couples cache allocation, expert batching and buffer lifetime while direct reads fill reusable GPU buffers without an extra user-space copy.The memory budget must also cover buffers whose GPU work has not finished, including when optional reads remain in flight.Source: §1 · p. 1
- English, Chinese and Python generation runs at 0.21–0.72 tokens per second with 4.495 GiB of reserved expert capacity.A 3,764-token retrieval prompt takes 309.0 seconds to the first token; these measurements use full prefill and local weights.Source: §1 · p. 2
- 21.0% higher aggregate decode throughput comes from adding next-layer speculation over resident-expert overlap, with 5.4% more expert data read.The matched comparison restores the same model state, warmup and continuation before each schedule.Source: §1 · p. 2
- The paper provides an implementation and measured account of SSD-backed execution for a 750B-class MoE on one 24 GB laptop.It also provides the implementation, converted weight layout and raw measurements for reproduction and extension.Source: §1 · p. 2
24 GB unified memory, one GPU
Figure 1 depicts SSD-backed execution: the full checkpoint stays on SSD while memory holds frequently used weights and changing experts. Routed records enter reusable buffers for GPU computation, while Engram lookup uses a separate row cache.
24 GB unified memory, one GPU
- The full checkpoint resides on the SSD, while memory holds frequently used weights and a changing set of experts.The figure distinguishes full model storage from the working set held in memory.Source: Figure 1 · p. 2
- The router determines which expert records enter the buffers, and the GPU consumes those records in place.Source: Figure 1 · p. 2
- Engram lookup follows a separate row cache rather than the routed-expert buffer path.Source: Figure 1 · p. 2
2 FROM MODEL SIZE TO WORKING SET
The model’s stored size is much larger than its active per-token working set: routed experts and Engram rows are selected as needed, while other weights remain resident. On a single-GPU machine, this makes storage movement central to execution.
2 FROM MODEL SIZE TO WORKING SET
- 15,360 routed experts are stored in the checkpoint, but a decoding token selects 240 across the model.Their weights amount to several GiB, and a cache can avoid reading some of them again.Source: §2 · p. 2
- About 7.6 GiB of weights for attention, routing, shared experts and the output head remain resident.This leaves room for expert reuse and temporary computation within physical memory.Source: §2 · p. 2
- Engram tables occupy 114.4 GiB after conversion, but each lookup reads only rows addressed by current tokens.Compressed sparse attention likewise stores a compressed history and uses an indexer to choose positions.Source: §2 · p. 2
- On a single-GPU machine, the system brings experts to the token rather than sending token activations to devices that already hold experts.This reverses the direction of data movement and makes the storage path central to the design.Source: §2 · p. 2
- A larger expert cache avoids more storage reads but takes memory from activations, sequence state and the operating system.More simultaneous reads may keep the SSD busy while leaving more buffers unavailable for reuse.Source: §2 · pp. 2–3
3 KEEPING THE TOKEN MOVING
DeepInfer-Mac moves routed experts from SSD into reusable GPU buffers and coordinates cache capacity with in-flight GPU work. Next-layer reads can provide a longer head start, but speculative reads may consume bandwidth without reducing latency.
3.1 A DIRECT PATH FROM SSD TO GPU
- Direct SSD reads fill preallocated GPU arrays with quantized expert records in the layout consumed by MLX, avoiding an extra user-space staging copy.The GPU later uses the same memory, and uncached reads avoid a second copy in the operating system’s page cache.Source: §3.1 · p. 3
- Resident experts can be computed while missing experts arrive, then missing experts pass through the same gathered matrix operations.This preserves the selected experts while changing when their work is issued.Source: §3.1 · p. 3
3.2 MEMORY THAT CAN ACTUALLY BE REUSED
- Cache capacity must protect experts until their GPU work is safe; demand acquisition protects hits and newly loaded experts together.Optional reads are admitted only while enough slots remain for the next demand group, and least-recently-used replacement saves a full expert read on a hit.Source: §3.2 · p. 3
3.3 THE LIMITS OF READING AHEAD
- Expert-byte reuse and transfer-time overlap are separate opportunities for reducing storage demand or hiding its delay.A layer must run its router before missing experts can be read and computed, limiting what storage bandwidth alone can remove.Source: §3.3 · pp. 3–4
- Next-layer predictions are issued after current demand reads, with low insertion priority, but unnecessary or lingering reads can still contend with later work.Demand-first admission does not guarantee lower latency; the paper compares speculation with overlap of resident-expert computation and missing reads.Source: §3.3 · p. 4
4 EXECUTING THE COMPLETE MODEL
DeepInfer-Mac implements the reference model’s compressed attention, indexers, Engram lookup and Hyper-Connections in MLX, with shared prefill and decode paths. It uses quantized weights and cache-bounded batching for long prompts, and validates implementation behavior on exercised cases without claiming equivalence to the quantized release.
4 EXECUTING THE COMPLETE MODEL
- The MLX implementation covers compressed attention, indexers, Engram lookup and Hyper-Connections, using a common forward path for prefill and decode.A streaming converter writes the final checkpoint one tensor at a time without first storing a second complete copy of the released model.Source: §4 · p. 4
- Routed experts use layer-selected affine quantization at two to four bits per weight, while Engram tables use four-bit rows.Dense FP8 tensors retain their values in MLX’s MXFP8 representation, but the checkpoint is lossy overall; per-layer precision trades reconstruction error against storage and read volume.Source: §4 · p. 4
- Full prefill batches experts to fit the cache budget; optional bounded replay omits much of the later-layer work on older prompt tokens as an approximation.End-to-end measurements use full prefill, while prefix reuse and replay remain available for applications with different latency tradeoffs.Source: §4 · p. 4
- Tests compare small configurations with the official PyTorch reference and exercise attention, routing, state updates, vision encoding and buffer-lifetime edge cases.These checks establish behavior for exercised cases, not equivalence between the quantized checkpoint and the release; English, Chinese, code and retrieval requests test the complete text path with real weights.Source: §4 · p. 4
5 WHAT THE LAPTOP CAN DO
The evaluation measures end-to-end generation and matched decode schedules on a shared M4 laptop, using a converted checkpoint, local Engram tables, and full prefill. It reports the memory footprint, prompt-startup cost, scheduling tradeoff, and limits of the correctness evidence.
5 WHAT THE LAPTOP CAN DO
- The end-to-end and matched schedule runs use the M4 laptop, local Engram tables, full prefill, and the same converted checkpoint; other desktop applications remained open.The shared-machine measurements are not an isolated hardware benchmark.Source: §5 · p. 4
5.1 GENERATION WITHIN THE MEMORY BUDGET
- Across requests, the largest sampled process footprint is 16.73 GiB, versus a 337.6 GiB checkpoint; the footprint includes more than the expert cache and is not an exact peak.Host swap was active before loading and grew during the suite, so the experiment does not establish swap-free operation.Source: §5.1 · p. 5, Figure 2 · p. 5
- Long prompts make prefill costly because they can visit most experts and must also build attention state, producing substantial startup latency despite a small resident working set.The retrieval requests use full prefill rather than hiding this cost with the optional bounded-replay approximation.Source: §5.1 · pp. 5–6
5.2 TRADING STORAGE TRAFFIC FOR LATENCY
- Next-layer speculation raises aggregate throughput from 0.92 to 1.11 tokens/s, a 21.0% increase over resident-expert overlap, while expert payload grows by 5.4%.It is faster in all nine matched blocks, but the measured schedules do not rank performance across SSDs, cache sizes, or concurrent workloads.Source: §5.2 · p. 6, Table 3 · p. 6, Figure 3 · p. 6
5.3 WHAT THE CORRECTNESS EVIDENCE COVERS
- The implementation passes 78 tests and matched runs preserve measured output tokens, but neither establishes the quality loss from low-bit conversion.The older teacher-forced comparison mixes revision and compression effects; a same-revision, broader quality evaluation remains necessary.Source: §5.3 · p. 7
6 WHEN COMPUTATION BECOMES VISIBLE
The indexer is a significant cost for long prompts, where it scores many query–key pairs before selecting keys. The paper tests faster arithmetic and score screening, with local speedups but limited scope for end-to-end gains.
6 WHEN COMPUTATION BECOMES VISIBLE
- The indexer scores many query–key pairs before selecting a small set, and its FP4 inputs with power-of-two scales permit faster arithmetic without another quantization step.Source: §6 · p. 7
- For 128-element dot products, BF16 multiplication with FP32 accumulation matches the reference FP32 final BF16 value under bounded block scales and no underflow or overflow.The runtime does not yet enforce the condition with a guard and fallback for every input.Source: §6 · p. 7
- 1.5–1.9 times faster matrix multiplication in measured blocks, with fusion of the subsequent head reduction removing repeated passes over intermediate scores.The BF16 path matched 1.078 billion scores on captured inputs from a 32K-token prompt.Source: §6 · p. 7
- DoSA screening saved only a modest fraction of arithmetic on captured queries, and its heuristic BF16 allowance is not a sound general bound, so screening remains disabled.The model’s existing candidate selection leaves a smaller domain to prune.Source: §6 · p. 7
7 RELATED WORK
Prior work addresses storage-backed inference, expert caching and prediction, Apple-silicon execution, and sparse-attention scoring. DeepInfer-Mac’s contribution is an auditable 24 GB case study, not a cross-system benchmark.
7 RELATED WORK
- Storage-backed inference has used device/host/disk scheduling, selective weight loading, or CPU computation for less-frequent neurons [Sheng et al., 2023; Song et al., 2024b].DeepInfer-Mac realizes and measures this broader strategy for a 750B-class model on a 24 GB unified-memory laptop.Source: §7 · p. 7
- Prior MoE systems use expert caching, prediction, prefetching, altered routing, or CPU expert computation [Eliseev & Mazur, 2023; Xue et al., 2024; Song et al., 2024a; Hwang et al., 2024; Kamahori et al., 2025].DeepInfer-Mac couples allocations for outstanding reads and GPU work within the single-sequence SSD setting.Source: §7 · p. 7
- Existing Apple-silicon projects provide DeepSeek-V4-generation engines and ports, while DeepInfer-Mac adds an auditable account of resource use and scheduling at 24 GB.A common cross-system benchmark remains future work.Source: §7 · p. 8
- GVR, LiteTopK, and DoSA reduce sparse-attention selection or scoring work [Cheng et al., 2026; Yin et al., 2026; Meng & Fang, 2026], while the measured BF16 path uses the model inputs’ discrete structure and an equality proof with explicit conditions.Accurate lower-precision matrix multiplication also has established precedents, including the Ozaki scheme [Mukunoki, 2025].Source: §7 · p. 8
8 DISCUSSION AND CONCLUSION
DeepInfer-Mac demonstrates local execution of a model much larger than laptop memory by separating the active working set from storage. Its operating point remains bounded by storage traffic, prompt latency, and a narrow evaluation scope.
8 DISCUSSION AND CONCLUSION
- A small laptop can execute DeepSeek-V4.1-Flash, a model whose stored parameters far exceed memory, when its active working set is small enough and remaining weights can be supplied from storage.The implementation runs the full text path and reports the memory and latency required.Source: §8 · p. 8
- Local access comes with storage traffic and substantial waits for longer prompts; retaining useful experts can avoid reads, while wrong predictions can consume memory and bandwidth and delay the next layer.The runtime must account for both effects within the same budget.Source: §8 · p. 8
- The evidence covers one laptop, one converted checkpoint, and one sequence at a time; it does not provide a broad quality benchmark, and host swap was active.The low-bit conversion and optional bounded replay need separate quality evaluation; bounded replay was disabled in these runs.Source: §8 · p. 8
AI USE STATEMENT
The authors disclose AI assistance with code, experiments, and manuscript drafting, while taking responsibility for the final manuscript and implementation.
AI USE STATEMENT
- Generative AI tools assisted with code, experiments, and manuscript drafting; the author takes responsibility for the final manuscript and implementation.The accompanying logs and tests document the checks performed.Source: AI USE STATEMENT · p. 8
REPRODUCIBILITY STATEMENT
The appendix documents the stored model, allocation guarantees, indexer optimizations, and reproducibility materials. It also records the limits of the capacity guarantee and the evidence supporting the implementation.
REPRODUCIBILITY STATEMENT
- The archive includes the implementation, converter, regression tests, measurement logs, and source fingerprints; saved measurements generate the tables and plots.Reference tests require the official source through DSV41_REF; without it, those tests are skipped.Source: REPRODUCIBILITY STATEMENT · p. 8
A.1 PARAMETER ACCOUNTING
- The audited model contains 748.495 billion weights, while the on-disk total is 337.6 GiB including optional components and quantization metadata.The 750B-class label avoids counting 6.144 billion Engram scale entries as parameters.Source: §A.1 · p. 11
A.2 STORED REPRESENTATION
- Expert records use layer-dependent 2–4-bit affine quantization, with 3 bits for the first four layers and selected sensitive layers, and 4 bits for the last layer.Table 4 reports expert bit widths and record sizes; DSpark draft experts use 2-bit gate/up and 3-bit down projections.Source: §A.2 · p. 11, Table 4 · p. 11
B PER-CASE API AGREEMENT
- The historical API comparison mixes a September 10 API snapshot with an October 17 checkpoint, so it cannot isolate quantization loss.The table reports 123/141 all-token agreement (87.2%) and 111/129 non-retrieval agreement (86.0%).Source: Table 6 · p. 12
C CAPACITY AND BUFFER LIFETIME
- The global store plan sizes heterogeneous records and prefill batches against a requested budget, while leases, pins, and read admission protect demand allocations and live GPU work.The guarantee covers capacity and buffer lifetime, not I/O latency, cancellation of SSD reads, RSS, or displacement of unpinned cache entries.Source: C CAPACITY AND BUFFER LIFETIME · p. 12
D INDEXER ARITHMETIC AND SCREENING
- The indexer scores visible query–key pairs and keeps the best K; a BF16 matmul can reproduce the reference rounded dot product under stated scale and overflow conditions.On captured inputs from a real 32K-token prompt, the BF16 version reproduced every one of about a billion scores; the fused head-reduction kernel matched the array version on tested inputs.Source: D INDEXER ARITHMETIC AND SCREENING · p. 13, §D.1 · p. 13
D.1 FASTER SCORING WITH THE SAME RESULTS
- A fixed 64-bit score-and-position key makes Top-K tie handling deterministic, selecting the earlier position when scores tie.NaN scores are mapped to negative infinity before selection.Source: §D.1 · p. 14
D.2 SKIPPING KEYS: AN EXPLORATORY SCREENING STUDY
The screening study finds only modest captured-input savings, with no general exactness guarantee, and leaves screening disabled. The appended measurement and provenance material defines the scope of these findings and related implementation evidence.
D.2 SKIPPING KEYS: AN EXPLORATORY SCREENING STUDY
- The BF16 slack used in the captured-query heuristic is not a certificate: a cited example exceeds its upper bound, and the implementation has no general exactness guarantee.The reported costs count fully scored seed keys, bound dot products for other eligible keys, and remaining heads for surviving keys.Source: §D.2 · p. 14
- A bound using about two thirds of the heads saves only a tenth to a fifth of indexer work before GPU compaction overhead, so screening remains disabled.The threshold is low because K is a large fraction of keys, and candidate selection already removes many keys; 128K-token contexts could change the picture, but prefill takes hours on this machine.Source: §D.2 · p. 14, Table 8 · p. 15
D.3 SCREENING MEASUREMENTS
- Table 8 reports each layer’s best bound configuration averaged over 17 queries, with up to 4096 recent eligible keys as the seed set.Its heuristic BF16 allowance makes these observed arithmetic costs, not certified pruning results.Source: §D.3 · p. 15, Table 8 · p. 15
D.4 EXACT ACCUMULATION FOR FP4 INPUTS
- For FP4 inputs, the BF16 matrix multiplication with FP32 accumulation and one final rounding matches the reference when scale-ratio, normal-range, and no-overflow conditions hold.On captured inputs, scale exponents were roughly −8 to 4; near-zero activations can violate the normal-range assumption.Source: §D.4 · p. 16
D.5 TOP-K ORDER KEY
- The 64-bit Top-K key orders higher scores first and earlier positions first on ties, with NaNs mapped to negative infinity.The position component applies for positions j < 2^31.Source: §D.5 · p. 16
E.1 FRESH END-TO-END REQUESTS
- The fresh generation suite uses full prefill and measures first-token latency, decode throughput, and sampled process memory; shared-desktop swap measurements are not model-attributable.The suite uses three short requests and two retrieval prompts; sampled maxima can miss short-lived peaks.Source: §E.1 · p. 16
E.2 MATCHED SCHEDULES
- Matched schedules restore the same prompt state, clear the cache, replay warmup, and consume identical continuations; read volume includes speculative submissions.The workload measures 324 tokens total, and throughput reflects forward-call latency rather than optional work completed outside timed calls.Source: §E.2 · pp. 16–17
E.3 COMMANDS AND PROVENANCE
- The archived logs retain environment versions, configuration, metadata hashes, buffer capacity, and source fingerprints for reproducing measurements.Figures and tables are generated directly from the saved files.Source: §E.3 · p. 17