Source-linked AI summary
FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution
Shuo Yang, Xiaoze Fan, Melissa Pan, Haocheng Xi, Zhe Wang, Shanlin Sun, Kurt Keutzer, Song Han, Matei Zaharia, Chenfeng Xu, Ion Stoica
TL;DR
Serving frontier open-weight models still largely assumes costly datacenter infrastructure, while heterogeneous, shared edge resources complicate local inference. FreeToken adapts model state and execution across CPU, GPU, host memory, and interconnect, sustaining 77–83 tok/s on Qwen3.6 and 22–25 tok/s on DeepSeek-V4-Flash while outperforming edge-serving baselines.
Problem
Frontier open-weight models still largely require costly datacenter-class GPU clusters, while heterogeneous, shared edge resources complicate local serving.
Method
FreeToken treats CPU, GPU, host memory, and interconnect as a unified platform, elastically adapting expert residency, execution, and memory to workloads and hardware.
Results
77–83 tok/s on Qwen3.6 and 22–25 tok/s on DeepSeek-V4-Flash, reaching 1.8–2.3× and 1.5–1.9× the strongest baselines, respectively.
Takeaways & Limitations
FreeToken enables personal hardware to serve models from 35B to 753B parameters across more than 20 MoE models and diverse edge-to-workstation systems.
Abstract
from arXiv · showhide
Frontier open-weight models are increasingly available, but serving them still largely assumes datacenter infrastructure. We present FreeToken, an edge-native MoE serving system that treats a personal machine not as a small GPU, but as a unified, elastic inference platform. FreeToken co-designs the full serving stack, including model layout and loading, expert residency, CPU--GPU execution, agentic state reuse, and runtime memory management, around two realities of local AI: agent workloads continuously change their execution pattern, and edge hardware exposes heterogeneous resources whose balance differs from machine to machine. Rather than committing to a fixed offloading strategy, FreeToken continuously maps computation and model state onto the resources actually available. FreeToken supports more than 20 MoE models and real coding and tool-using agents across hardware ranging from an 8GB laptop GPU to a single workstation GPU. More importantly, it changes what these machines can practically serve, from a 35B model on a laptop to a 284B model on a gaming desktop and the 753B GLM-5.2 on a single workstation GPU. FreeToken turns open weights into deployable local software, making the machines users already own a practical platform for frontier-scale intelligence. We release the system at flashml.ai.
1 Introduction
FreeToken addresses the gap between frontier open-weight models and affordable local serving by treating heterogeneous consumer hardware as a unified, elastic inference platform. It co-designs bandwidth-adaptive execution and runtime resource management for the distinct sparsity, memory, and workload challenges of MoE inference.
- Motivation: More than one hundred million consumer machines already contain discrete GPUs, but existing systems lack a serving layer that unifies and automatically maps their heterogeneous resources.These machines include gaming desktops, workstations, and high-performance laptops, representing a large pool of underutilized compute.
- MoE serving challenge: MoE sparsity reduces per-token computation but leaves the complete expert pool in memory, forcing inactive experts beyond VRAM into host memory or secondary storage.DeepSeek-V4-Flash activates 6 of 256 routed experts per token while retaining a much larger full model footprint.
- MoE serving challenge: Prefill becomes effectively dense because long prompts collectively route through most experts, causing repeated expert streaming and severe memory-movement costs in growing agentic contexts.The problem is particularly severe for agentic tool-calling workloads with long, continuously growing contexts.
- MoE serving challenge: Decode has sparse token-level routes but repeated cache misses, while static placement and prediction do not decide how unavoidable misses should use PCIe transfer, GPU execution, or CPU execution.The missing policy must account for changing routes and divide work across the available execution paths.
- FreeToken design: FreeToken uses bandwidth-adaptive execution, double-buffering expert movement during prefill and applying a q⋆ policy during decode to partition bandwidth-dependent work.PCIe transfer and direct CPU expert execution draw from the same host-memory bandwidth, motivating fine-grained allocation.
- Evaluation scope: FreeToken supports more than 20 MoE models and evaluates representative frontier models across six machines, four real agentic workloads, and comparisons with four existing serving systems.The evaluated hardware ranges from an 8 GB RTX 4060 laptop to a single RTX PRO 6000 workstation.
2 Challenges in Edge MoE Serving
Edge MoE serving is constrained by prefill overheads, decode-time expert misses, and rapidly changing local resources. Existing engines struggle because expert traffic shifts across tokens, CPU bandwidth is far below GPU memory bandwidth, and serving competes with other applications.
- Prefill challenges: Prefill increases TTFT because expert transfer scales with the full model and agentic sessions repeatedly recompute context.Decode activates few experts per token, but prefill processes thousands of tokens per layer and can stream nearly the complete expert pool.
- Decode challenges: Static expert placement misses routed traffic because routing changes with every token and workload.Existing engines assign experts at load or prefill time, so frozen placement captures only a small fraction of later routed traffic.
- Decode challenges: 50 GB/s versus 1–1.8 TB/s leaves consumer CPUs unable to carry decode at GPU-comparable speed.The cited figures compare peak bandwidth for dual-channel DDR4 with the on-package memory bandwidth of a single RTX 4090 or 5090.
- Decode challenges: The optimal split between CPU execution and PCIe transfers is hardware-specific and must be determined quantitatively on the deployed machine.Transfer-only and CPU-only policies can each leave resources underused, while hardware configurations such as an RTX 4060 laptop and RTX 5090 desktop differ substantially.
- Dynamic edge resources: Dynamic VRAM availability and growing KV-cache demand continually change the best model-state split during serving.Desktop applications can claim gigabytes of VRAM, while agentic sessions accumulate context and the expert working set remains roughly fixed.
- Dynamic edge resources: 20 seconds is required just to read roughly 140 GB from a 7 GB/s NVMe drive in an FP4 DeepSeek-V4-Flash deployment.This startup cost recurs when users reopen the engine, free the machine, or switch models.
3 FreeToken Design
FreeToken designs edge MoE serving as an elastic system that coordinates CPU-resident experts, a shared GPU cache, and phase-specific execution. It hides prefill transfers, preserves reusable agent state across edits, adapts decode misses to measured bandwidth, and reconfigures runtime memory as hardware availability changes.
- 3 FreeToken Design: FreeToken keeps complete routed-expert weights in a CPU-resident source-of-truth pool and uses remaining GPU memory as a shared elastic cache across MoE layers.Each cache slot stores all tensors needed for one layer–expert pair, so residency, lookup, and execution share a logical (layer, expert) identifier.
- 3.1 Prefill: During prefill, full-layer double buffering streams layer l+1 while the GPU computes layer l, hiding expert transfer behind computation.The design loads complete expert sets because prefill activates nearly the entire expert set of every layer.
- 3.1 Prefill: Semantic anchors preserve recurrent checkpoints and full-attention KV prefixes across context edits, allowing re-prefill of only the genuinely new suffix.Anchors correspond to special-token boundaries such as thinking segments, tool calls and outputs, and conversation turns; checkpoint slots use independent LRU eviction.
- 3.2 Decode: During decode, FreeToken routes cache hits on the GPU and divides missing experts between cache fills and CPU execution according to measured pinned transfer bandwidth BP and host processing bandwidth BH.The two branches run concurrently: fills update GPU residency, while CPU-executed experts remain in the CPU pool and do not alter residency.
- 3.2 Decode: The bandwidth-adaptive split preserves exact MoE outputs by merging CPU and GPU partial sums, with exposed latency determined by the slower concurrent branch.As BH approaches BP, q⋆ approaches the total miss count m, reducing execution to pure on-demand cache fill.
- 3.3 Runtime Memory Management: FreeToken dynamically reallocates GPU memory between KV pages and complete-expert slots, rebuilding the expert cache at scheduler safe points without restarting or reloading the CPU expert pool.This addresses fluctuating VRAM availability and changing memory demand as contexts grow.
4 Implementation
FreeToken implements bandwidth-adaptive MoE serving through GPU-resident dynamic cache control and graph-captured heterogeneous execution. It also unifies expert storage and adapts kernels and CPU backends to model, hardware, and platform constraints.
- Graph-compatible expert cache: GPU-resident routing control keeps expert-cache decisions inside statically captured graphs, avoiding device synchronization at every MoE layer.Fixed-shape work buffers and device-resident data represent routing-dependent control.
- Graph-compatible expert cache: A single GPU kernel deduplicates routed experts, computes bandwidth-based fetch count q, selects eviction victims, and rewrites logical IDs into physical slots or CPU assignments.Shared expert-to-slot mappings enable one fixed-shape launch across all expert banks, with valid counts masking unused work.
- Graph-compatible expert cache: Graph-captured CPU execution replays device copies, host submission, concurrent GPU work, synchronization, and result copies without per-token Python scheduling.FreeToken prepares stable pinned I/O buffers and persistent task descriptors for each supported decode batch size.
- Storage and platform machinery: Expert banks flatten layer–expert identifiers so GPU kernels and the CPU executor share one logical expert identity across physical checkpoint formats.The FTW format normalizes model-specific checkpoint layouts and accelerates loading.
- Storage and platform machinery: At load time, FreeToken selects compatible GPU kernels and CPU SIMD implementations, falling back to a pure-CPU MoE backend when complete expert pinning or DMA registration is unavailable.The fallback keeps expert weights in pageable host memory.
5 Evaluation
FreeToken is evaluated on real agentic workloads across six heterogeneous GPU systems, using MoE models whose expert pools exceed VRAM and bit-exact aligned baselines. It delivers higher and more stable decode throughput, lower tail latency, and consistent gains across hardware through pipelined prefill and expert-cache locality.
- 5.1 Setup: FreeToken is evaluated on four agentic workloads, two MoE models, six discrete-GPU systems, and actively maintained edge-serving baselines with aligned weight formats.The evaluated systems range from consumer hardware to a workstation-class RTX PRO 6000 Blackwell with 96 GB.
- 5.2 Main results: 77–83 tok/s on Qwen3.6 and 22–25 tok/s on DSV4-Flash are 1.8–2.3× and 1.5–1.9× the strongest baseline, respectively.FreeToken remains within 12% of its single-turn W1 rate across the three agent workloads, whereas KTransformers on DSV4-Flash loses 31% at W2.
- 5.2 Main results: 44 s is FreeToken’s worst-turn TTFT ceiling in every cell, while every baseline exceeds 150 s somewhere, including 232 s for llama.cpp, 179 s for Ollama, and 946 s for KTransformers.FreeToken has the lowest mean TTFT in five of six multi-turn cells; KTransformers favors Qwen3.6×W3, while llama.cpp favors W1’s short isolated prompts.
- 5.3 Mechanism analysis: 1.19–1.22 s per 8,192-token prefill chunk matches streaming the 64.4 GB expert pool at 52.7 GB/s, hiding expert computation behind transfer and reaching 6.7k tok/s at 16k tokens.Removing the second buffer reduces throughput by 19% at 4k tokens and 25% at 8k.
- 5.3 Mechanism analysis: 16% and 39% decode-time expert-read miss rates for FreeToken’s global LRU beat KTransformers’ 41% and 59% at equal cache capacity.The replay uses identical routing traces from all four workloads at RTX 5090 capacity: 37% of Qwen3.6’s expert pool and 11% of DSV4-Flash’s.
- 5.3 Mechanism analysis: 1.3×–2.1× gains over the strongest baseline hold across five consumer systems, while the RTX 4060 laptop sustains 39.3 tok/s on 8 GB, reaching 92% of the RTX 4090 rate.The reported gains are 1.3× on the RTX 3090 and 4090, 1.9× on the RTX 5090 server, 2.1× on the RTX 5090 desktop, and 1.8× on the RTX 4060 laptop.
6 Related Work
FreeToken builds on prior work in expert caching, hybrid CPU–GPU execution, GPU-centric serving, and hierarchical memory management. It differs by targeting dynamic, multi-turn agentic workloads with cross-request prefix reuse and execution that remains compatible with CUDA Graphs.
- Expert offloading and caching: Expert offloading commonly keeps the full MoE expert pool in host memory or on disk while caching a subset on the GPU.EdgeMoE established this design for on-device inference; Mixtral-offloading added LRU caching and speculative prefetching, while MoE-Infinity traces request-level activation patterns.
- Hybrid CPU–GPU execution: Prior hybrid CPU–GPU systems use the CPU for computation or weight storage, including layer-granular streaming, activation-based neuron splitting, and whole-layer assignment.FlexGen and DeepSpeed-Inference target throughput-oriented batch inference; PowerInfer requires ReLU-family sparsity and learned predictors; llama.cpp and Ollama assign whole layers.
- Hybrid CPU–GPU execution: Existing hybrid systems target single-shot, short-prompt inference and generally cannot preserve CUDA Graph execution when host-side scheduling or per-layer synchronization is required.The passage also notes that these systems are often evaluated on unquantized weights.
- Hybrid CPU–GPU execution: FreeToken distinguishes itself by supporting cross-request prefix reuse for multi-turn agentic sessions that repeatedly re-enter prefill during tool-calling turns.The cited comparison states that prior systems do not provide this reuse.
- Serving infrastructure and hierarchical memory: FreeToken builds on vLLM, SGLang, and FlashInfer for GPU-centric serving, while SGLang HiCache tiers KV cache across GPU, host, and remote storage to preserve long-session prefix reuse.FlashInfer provides precedent for dynamic behavior inside CUDA Graph capture.
7 Conclusion
FreeToken is an edge-native serving system for frontier-scale MoE models on personal hardware. It treats local inference as a resource-orchestration problem, unifying the GPU, CPU, host memory, and interconnect while adapting model state and execution to agentic workloads and available hardware.
- 7 Conclusion: FreeToken enables serving frontier-scale MoE models on personal hardware through an edge-native system.The paper presents FreeToken as a serving system designed for personal machines.
- 7 Conclusion: FreeToken reframes local inference from GPU fit to coordinated orchestration once sparse activation makes model computation feasible.Its central observation is that sparse activation shifts the challenge toward system coordination.
- 7 Conclusion: FreeToken unifies the GPU, CPU, host memory, and interconnect, adapting model state and execution to agentic workloads and their hardware.The system responds to changing workload structure and differences in the machines on which inference runs.