PaperlayerView exact revision on arXiv

Source-linked AI summary

NeuroPrefetcher: Storage-Aware Sparse LLM Inference via Delta Prefetching

Nobel Dhar, Md Romyull Islam, Xuechen Zhang, Gongjin Sun, Sahidul Islam, Bobin Deng, Kun Suo

arXiv:2608.22643v1cs.DCcs.LG

TL;DR

Edge LLM deployment is constrained when model size exceeds resident memory, making storage-backed weight movement part of the inference path. NeuroPrefetcher predicts sparse activity after layer 0 and prefetches only newly needed rows, achieving 7.9–12.0× speedup over llama.cpp on unified-memory edge hardware.

  • Problem

    The paper targets inference where the model remains larger than resident memory, a setting existing compression and offloading approaches are not designed to handle.

  • Method

    NeuroPrefetcher uses a shared post-layer-0 predictor to forecast downstream sparse activity and application-scheduled NVMe reads to prefetch only nonresident delta rows.

  • Results

    7.9–12.0× speedup over llama.cpp is achieved on real unified-memory edge hardware in the memory-constrained regime.

  • Takeaways & Limitations

    Delta reuse and explicit sparse-row movement make storage-backed sparse execution effective when the full model remains on storage throughout inference.

  • Takeaways & Limitations

    The approach is most useful when the model exceeds resident memory; after the crossover, sparse storage-backed execution provides less benefit.

Abstract

from arXiv · show

Deploying large language models on edge devices is increasingly limited by a widening gap between model size and available memory. Existing approaches such as quantization, smaller models, and offloading can raise the effective memory limit, but they still assume that the model can be compressed or partitioned to fit within some budget. We target the harder model-exceeds-memory setting, in which the model remains larger than resident memory throughout execution and storage becomes an active source of weights on the critical path. We observe that MLP activity during autoregressive decoding has strong temporal locality: approximately 82-85% of active neurons persist from one token to the next. This means that most sparse weights needed for the current token are already resident, and only the newly needed rows must be fetched from storage. We present NeuroPrefetcher, a storage-backed LLM inference system that exploits this property through predictive delta prefetching. After layer 0, a single GPU-resident predictor, occupying 2.86% of base model parameters, predicts sparse activity for all downstream MLP layers in one forward pass. The runtime compares these predictions against resident GPU buffers and issues application-scheduled NVMe reads only for incoming delta rows, replacing reactive operating-system demand paging with explicit, model-aware weight movement. On real unified-memory edge hardware, NeuroPrefetcher achieves 7.9-12.0x speedup over llama.cpp across constrained memory budgets.

1 Introduction

Edge deployment faces a widening model–memory gap, especially when models remain larger than resident memory and storage must supply weights during inference. NeuroPrefetcher addresses this regime with predictive delta prefetching, achieving substantial speedups over reactive paging.

  • Model size has grown roughly 10× per generation while edge DRAM capacity has grown only about 2×.
  • When the model exceeds resident memory, storage becomes an active weight tier whose movement directly affects inference performance.
  • At mem=14G, llama.cpp generates 11,453 major page faults per second, with the CPU blocked on I/O 77% of the time and GPU utilization averaging 3%.
  • NeuroPrefetcher predicts downstream sparse activity after layer 0, compares it with resident rows, and fetches only incoming delta rows before execution.A single GPU-resident predictor exposes the sparse-layer I/O plan in one forward pass, replacing reactive demand paging with application-scheduled movement.
  • NeuroPrefetcher achieves 7.9–12.0× speedup over llama.cpp on constrained unified-memory edge hardware.The evaluation uses Jetson AGX Orin with Mistral-7B-v0.1 and Llama-3-8B; at mem=14G, throughput reaches 3.22 tok/s versus 0.34 tok/s.
  • 82–85% of the previous token’s active set is reused, while the predictor preserves 92–96% of dense accuracy without per-task retraining.

2 Related Work

Existing approaches reduce footprint, exploit sparsity, or offload weights, but generally assume residency or react when execution reaches missing weights. NeuroPrefetcher instead uses sparsity to schedule incremental NVMe movement before sparse execution.

  • Quantization, pruning, distillation, and compact model design reduce footprint but assume the deployed model can be made resident.Quantization is described as orthogonal to NeuroPrefetcher, which targets cases where even the quantized model exceeds memory.
  • Prior sparsity methods mainly reduce computation, whereas NeuroPrefetcher uses predicted active neurons to determine which MLP rows cross the NVMe boundary.
  • Offloading systems such as FlexGen and llama.cpp move weights reactively through streaming or memory-mapped paging; NeuroPrefetcher schedules movement before sparse execution.

3 Design of NeuroPrefetcher

NeuroPrefetcher combines offline sparse-centroid preparation, a shared runtime predictor, and storage-backed execution that fetches newly needed weights.

  • NeuroPrefetcher combines offline sparse centroid preparation, runtime centroid selection, and storage-backed execution that fetches newly needed weights.These components are used in that order.

3.1 System Overview

The system keeps early layers dense while later MLP layers use bounded GPU buffers backed by NVMe, with predicted activity determining each token’s fetch delta.

  • The first d transformer layers remain dense, while the remaining N−d layers use GPU buffers of K_max neuron rows backed by NVMe.Embeddings, attention projections, normalization parameters, the predictor, and centroid table remain resident.
  • After layer 0, the predictor emits gate and up centroid IDs for later layers, whose predicted active sets are compared with resident rows.The difference identifies the rows newly required for the current token.
  • A background thread issues batched NVMe reads for each delta before sparse execution proceeds.This makes predicted activity drive lookahead weight movement.

3.2 Offline Sparse Centroid Preparation

Offline preparation converts dense MLP activations into sparse masks and clusters similar masks into a compact centroid vocabulary for runtime prediction.

  • Thresholding and clustering convert dense MLP activation patterns into a compact runtime representation.The vocabulary avoids predicting one binary state for every downstream neuron.
  • Activation Thresholding for Sparsity Enforcement: Per-layer percentile thresholds remove low-magnitude MLP activations without retraining, producing sparse activation patterns.Higher α removes more neurons and reduces storage traffic, while lower α preserves more neurons and quality.
  • Clustering of Sparse Activation Patterns: Thresholded masks are clustered so similar active-neuron sets share sparse binary centroids.More clusters preserve finer-grained patterns but make prediction harder.
  • Clustering of Sparse Activation Patterns: Each centroid aggregates cluster-member masks and is pruned to the target sparsity level, reducing runtime prediction to one centroid choice per layer.The resulting centroids remain sparse by construction.

3.3 Shared Activation Predictor

A shared GPU-resident predictor uses layer 0 features to select sparse centroids for all downstream MLP layers, with a fixed runtime cost relative to the base model.

  • The predictor runs once per token and outputs 62 centroid identifiers for gate and up projections across layers 1 through 31.Down projections remain dense because sparsifying their 4,096-neuron dimension noticeably hurts output quality.
  • Four continuous layer 0 semantic features and a gate-activation centroid encode the token’s early state and activation structure.The centroid code is cheaper than feeding the full 14,336-dimensional gate vector.
  • Architecture: The architecture projects semantic features, combines them with a gate centroid embedding, and processes the result through a residual MLP trunk.The trunk combines semantic and structural signals into a shared feature space.
  • Architecture: Two groups of 31 independent linear heads predict gate and up centroid logits, selecting the highest-probability centroid for each layer.The logits span a centroid vocabulary of size K.
  • Training: The predictor is trained with cross-entropy over gate and up centroid IDs and remains fixed during runtime.It uses AdamW, distributed data parallelism, T=512, a four-block residual trunk, and K=512.
  • Predictor Cost: 206.8M parameters comprise 2.86% of the base model and occupy 414 MB in FP16.All reported speedups include this always-resident predictor cost.

3.4 Predictive Delta Prefetching

NeuroPrefetcher predicts downstream sparse activity, compares it with resident GPU rows, and updates each layer’s buffer using only newly active rows.

  • Predictive Delta Prefetching: The shared predictor identifies expected active neurons for each sparse layer, enabling incremental GPU-buffer updates instead of reloading each active set.Gate and up centroid masks are intersected to form the active set used by the SwiGLU MLP projections.
  • Predictive Delta Prefetching: Incoming rows are fetched from NVMe, while outgoing rows release buffer slots for reuse.Each active set has fixed size K_max, so incoming and outgoing rows closely match and buffer compaction is avoided.
  • Predictive Delta Prefetching: A per-layer slot map tracks resident neurons and their buffer locations, allowing sparse kernels to find active rows without requiring sorted buffer order.Correctness depends on row presence and slot address rather than buffer ordering.
  • Predictive Delta Prefetching: The runtime batches NVMe reads for incoming rows across sparse layers, initializes the first token with full predicted active sets, and uses deltas thereafter.After cold start, the incoming-row ratio drops quickly and stabilizes in steady state.

3.5 I/O Path

The I/O path aligns storage layout with sparse-neuron execution and transfers incoming rows asynchronously into GPU-resident projection buffers.

  • I/O Path: The runtime uses a model-aware sparse-row transfer path combining offline layout preparation, read coalescing, asynchronous I/O, coherent staging, and fused GPU scatter.On Jetson, mapped pinned staging handles unified-memory coherence.
  • I/O Path: Each 24.5 KiB record co-locates the gate, up, and downT projection rows for one intermediate neuron in a contiguous layer-major file.The transposed down projection gives all three rows a shared row-major shape and stride, so one record read supplies one neuron’s projection rows.
  • I/O Path: Incoming neuron IDs are sorted and adjacent IDs are coalesced into contiguous O_DIRECT io_uring reads with queue depth 256.The queue depth is intended to saturate the drive for this access pattern; deeper queues add no bandwidth for coalesced records.
  • I/O Path: A fused CUDA deinterleave-and-scatter kernel writes staged records into projection-specific buffers while its tail overlaps dense-prefix computation.If a token delta exceeds staging capacity, the runtime processes it in independent submit-and-scatter chunks.

4 Evaluation

Evaluation on unified-memory Jetson hardware tests NeuroPrefetcher across memory budgets, sparsity levels, partition choices, throughput, I/O, latency, and model quality. The system is most advantageous while the model exceeds resident memory, where delta reuse reduces storage traffic and improves throughput over llama.cpp.

  • Evaluation: Experiments run on a Jetson AGX Orin with 32 GiB unified LPDDR5 memory, an Ampere GPU, and a Samsung 990 PRO NVMe SSD.The SSD is rated for 7.4 GB/s sequential reads.
  • Evaluation: Seven system-memory limits from 11 GiB to 17 GiB are evaluated in 1 GiB steps to represent changing memory available to inference.The limits account for memory consumed by the operating system, display server, and CUDA runtime.
  • Evaluation: The study evaluates K_max = 5,500, 7,168, and 9,865, corresponding to 62%, 50%, and 31% sparsity, respectively.Throughput uses 2,048 generated WikiText-2 tokens; quality uses WikiText-2 perplexity and 10-shot normalized HellaSwag accuracy.
  • 4.3 Dense/Sparse Partitioning: At 62% sparsity, the dense-prefix sweep rises from 1.23 tok/s at d=2 to 1.60 tok/s at d=14, then falls to 0.19 tok/s at d=32.The fall occurs when most MLP layers are served through whole-layer NVMe reads.
  • 4.4 End-to-End Throughput: NeuroPrefetcher achieves 7.9–12.0× speedup over llama.cpp across 9.0–13.2 GiB CUDA-available memory, where llama.cpp sustains 0.2–0.3 tok/s.At 62% sparsity, NeuroPrefetcher reaches 1.7 tok/s at 9.0 GiB and 3.7 tok/s at 13.2 GiB; lower sparsity can trigger whole-layer reads at the tightest budget.
  • 4.5 I/O Reduction via Delta Reuse: At 62% sparsity, 82–85% of sparse suffix rows are reused across tokens, while per-token NVMe reads range from 103 MiB at 14.8 GiB to just under 1 GiB at 9.0 GiB.Tighter budgets shift more MLP layers from dense residency to sparse execution.
  • 4.6 Latency Breakdown: NVMe I/O accounts for 80–87% of per-token latency across memory levels, while GPU compute and predictor inference remain secondary costs.Total latency falls from 593 ms per token at 9.0 GiB to 113 ms at 14.8 GiB as more MLP layers remain dense and resident.
  • 4.7 Model Quality: At d=20, Mistral-7B retains roughly 95–96% and Llama-3-8B roughly 92–93% of dense HellaSwag accuracy, depending on sparsity.Increasing d improves retention, while higher sparsity and smaller dense prefixes support tighter memory budgets.

5 Conclusion

NeuroPrefetcher targets inference where LLMs exceed resident memory, using explicit sparse-row movement and delta reuse to accelerate edge inference. It achieves 7.9–12.0× speedup over llama.cpp, while NVMe I/O remains the dominant latency bottleneck.

  • 7.9–12.0× speedup over llama.cpp on memory-constrained unified-memory edge hardware demonstrates NeuroPrefetcher’s effectiveness.The system replaces page-fault-driven whole-weight movement with application-scheduled sparse-row reads.
  • 82–85% of each layer’s sparse rows remain resident across tokens, so only changed rows are fetched per token.
  • NVMe I/O dominates per-token latency even after delta prefetching, identifying storage throughput as the remaining bottleneck.Suggested future directions include deeper prefetch pipelines, stronger read coalescing, quantized sparse-row formats, and improved transfer–execution overlap.
  • NeuroPrefetcher is presented as the fastest distinct operational system for the model-exceeds-resident-memory setting.
Loading 2608.22643v1…