Source-linked AI summary
LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference
Renyuan Liu, Yuyang Leng, Kaiyan Liu, Yuzhou Zhong, Shaohan Hu, Chun-Fu, Chen, Peijun Zhao, Heechul Yun, Shuochao Yao
TL;DR
On-device sparse LLM inference must balance accurate, fresh-context decisions against early prediction for computation–I/O overlap. LeanStream progressively refines execution and loading priorities from partial GPU results. On mobile and embedded platforms, it reduces memory usage by 4.8× to 7.5× at prior work’s best throughput and improves token generation throughput by 1.6× to 2.1×.
Problem
Accurate sparse execution decisions need the latest context, but dependence on updated context limits I/O prefetching and pipelining.
Source: §1 · p. 1Method
LeanStream progressively updates computation priorities and data loading and retention decisions as intermediate results become available.
Source: §1 · p. 2Results
4.8× to 7.5× lower memory usage at prior work’s best throughput and 1.6× to 2.1× higher token generation throughput on mobile and embedded platforms.
Source: §1 · p. 3Takeaways & Limitations
LeanStream’s evaluation reports improved resource utilization and inference efficiency over prior approaches on mobile and embedded platforms.
Source: §1 · p. 3High-frequency coordination is constrained by synchronization overhead, and event-based primitives lack bidirectional host control and may require manual kernel partitioning.
Source: §1 · p. 2
Abstract
from arXiv · showhide
On-device LLM inference is attractive for privacy and responsiveness, but remains challenging on mobile and embedded devices because model weights far exceed available DRAM. Prior systems exploit activation sparsity and offload weights to SSD or flash storage, but face a fundamental systems trade-off: accurate sparse execution decisions require the latest context, whereas efficient computation-I/O overlap requires early prediction. As a result, existing designs either serialize execution or incur redundant weight fetches, extra computation, and large cache overheads. We present LeanStream, a streaming speculate-and-refine framework for efficient on-device LLM inference. LeanStream progressively refines computation, loading, and cache-retention priorities using partial GPU results, enabling fine-grained overlap between GPU execution and storage I/O. We implement LeanStream on both mobile and embedded platforms. Compared with prior on-device LLM inference systems, LeanStream reduces memory usage by 4.8$\times$ to 7.5$\times$ at the best throughput achieved by prior work, while further improving token generation throughput by 1.6$\times$ to 2.1$\times$.
1 Introduction
Activation sparsity can reduce on-device inference costs, but accurate decisions depend on fresh context while overlapping computation and I/O requires early prediction. LeanStream progressively refines execution and data-movement priorities from partial results, coordinating GPU work and storage transfers at fine granularity.
1 Introduction
- Activation sparsity can reduce memory and computation costs by up to 80%, but fresh-context decisions limit I/O prefetching and pipelining, while early speculation can fetch unnecessary weights and execute extra sub-models.Full-context prediction serializes compute and I/O; layer-wise speculation overlaps them but risks prediction errors and redundant I/O.Source: §1 · p. 1, Figure 1 · p. 2
- Fine-grained CPU, GPU, and I/O coordination must adapt online because frequent synchronization can sacrifice hardware parallelism and runtime I/O latency varies.LeanStream’s lightweight communication and data-management framework is designed to reduce synchronization overhead and metadata traffic.Source: §1 · pp. 2–3
- LeanStream starts with the freshest available context, then refines computation priorities and data loading and retention as intermediate results arrive.Fine-grained weight streaming lets the GPU execute available high-priority sub-models without waiting for the entire sparse layer weight set.Source: §1 · p. 2
- LeanStream stages MLP computation in priority order and accumulates each stage’s output with the residual state to refine next-layer neuron priorities.This replaces a fixed one-shot prediction from the residual state entering the preceding MLP.Source: Figure 2 · p. 3
- Stacked learnable hashing provides a low-latency, memory-efficient predictor for system control, using bitwise operations, in-register table lookups, and compact outputs.It is described as more expressive and lower-overhead in runtime and memory than conventional shallow-MLP controllers.Source: §1 · p. 3
- 4.8× to 7.5× lower memory usage at prior work’s best throughput, and 1.6× to 2.1× higher token generation throughput, across mobile and embedded platforms.Evaluations covered Mistral-7B, Llama2-7B, and Qwen2.5-7B on Jetson AGX Orin, Jetson AGX Xavier, and OnePlus 13.Source: §1 · p. 3
2 Motivation & Related Work
On-device sparse LLM inference must balance accurate, context-aware decisions against overlapping computation and I/O. LeanStream refines priorities progressively and adapts coordination to runtime conditions.
2.1 Challenges of on-Device LLM Inference
- On-device LLM deployment is limited by model weights exceeding available DRAM; exploiting dynamic activation sparsity can skip around 80% of unnecessary computation and reduce data movement with negligible accuracy loss [3] [6].Source: §2.1 · p. 3
- Full-context decisions serialize GPU computation and I/O, whereas early prediction overlaps them but can cause inaccurate predictions, redundant fetching, and extra execution.LeanStream addresses this tension by refining priorities as intermediate results become available.Source: §2.1 · pp. 3–4
- LeanStream reduces redundant data movement and computation by progressively refining computation, loading, and cache-retention priorities as intermediate results arrive.Earlier partial outputs also help refine priorities for the next MLP before the current MLP completes.Source: §2.1 · p. 4
- Increasing synchronization frequency sharply raises GPU execution time for a Mistral-7B MLP layer with 80% sparsity on Jetson AGX Orin.The overhead includes synchronization itself and kernel fragmentation from splitting monolithic kernels into smaller schedulable units.Source: §2.1 · p. 4
2.2 The Inefficiency of Static Coordination
- Cache miss rates vary across Mistral-7B layers and prompts under a 50% weight-cache budget, making static profiling unreliable for coordination.The timing relationship between I/O fetching and GPU computation shifts with runtime cache-miss behavior.Source: §2.2 · p. 4, Figure 4 · p. 4
2.3 Prediction Overhead
- A CPU two-layer MLP predictor can take longer than sparse GPU MLP computation on Jetson AGX Orin, and its cost grows with finer-grained coordination.The comparison measures latency for Mistral-7B with 80% sparsity; the predictor follows state-of-the-art designs [2, 4].Source: §2.3 · p. 4, Table 1 · p. 5
3 LeanStream Design
LeanStream overlaps GPU execution and I/O by continuously refining weight priorities and starting work as soon as required weights arrive. Its fine-grained control uses asynchronous, non-blocking coordination between GPU thread blocks and CPU-side I/O.
3.1 Overview
- Partial GPU results continuously reprioritize important weights for the next layer, while the GPU starts corresponding partial computation as soon as a subset of weights loads.This decouples computation from the rigid layer-by-layer I/O schedule and overlaps computation with I/O.Source: §3.1 · p. 5
3.2 Fine-Grained Streaming Control
- LeanStream uses a non-blocking, asymmetric, thread-block-level communication mechanism to coordinate GPU computation and I/O without kernel splitting.The mechanism uses the SoC's unified DRAM for direct inter-device coordination.Source: §3.2 · p. 5
3.2.1 Lightweight Coordination Primitives.
- GPU thread blocks check whether weights are loaded and publish partial results; the CPU can aggregate completed blocks to update I/O selection asynchronously.A single CPU update can respond to multiple GPU signals, and one I/O completion can resolve dependencies for multiple blocks.Source: §3.2.1 · pp. 5–6
3.2.2 Analysis of Coordination Overheads.
- Prioritizing weight fetches by real-time importance inherently disrupts sequential access patterns.The section analyzes the resulting I/O and computation overheads to motivate coordination scheduling.Source: §3.2.2 · p. 6
1 Function GPU_Kernel(layer_id, group_id)
The kernel code places barriers around sparse computation and signals block completion with an atomic store.
1 Function GPU_Kernel(layer_id, group_id)
10 Function I/O_Worker(layer_id, group_id, lane_id)
LeanStream combines adaptive streaming coordination with lightweight learned prediction to manage weight loading, computation, and cache retention. Its methods also organize weights for I/O and allow computation to use weights as they arrive.
10 Function I/O_Worker(layer_id, group_id, lane_id)
- Out-of-order weight access can increase I/O operations and reduce transfer sizes, so LeanStream optimizes physical weight placement using neuron co-activation patterns.The placement strategy is introduced to mitigate bandwidth loss from more frequent, smaller I/O requests.Source: §10 · p. 6
n TGPU
- For I/O-aware placement, LeanStream learns neuron embeddings from co-activation contexts, clusters them, and groups weights into blocks for each model.The default block groups contain three weights for Qwen2.5-7B and two for Mistral-7B and Llama2-7B.Source: n TGPU · p. 7
- SwiGLU hidden-neuron permutation invariance lets LeanStream append out-of-order weight blocks and execute them directly without re-indexing or reshuffling.The output sums over hidden neurons, so their internal order does not change the result.Source: n TGPU · p. 7
3.2.3 Fine-Grained Streaming Control Strategy.
- Stochastic MPC models runtime variation and selects a coordination plan offline; online, LeanStream applies only the current-layer plan through a lookup table.The plan balances coordination overhead against GPU waiting and delayed use of refined predictions.Source: §3.2.3 · pp. 7–8
3.3 Lightweight System Control with Stacked Learnable Hashing
- Stacked learnable hashing replaces standard neural primitives with learned indexing and lookup tables, targeting low-latency, memory-efficient CPU prediction for frequent system control.Its computation uses bitwise operations and in-register table accesses, with compact outputs.Source: §3.3 · p. 8
3.3.1 Stacked Learnable Hashing.
- Each learnable hashing layer computes a hash code and uses it to retrieve a learnable binary feature vector from a table.The hash code uses hyperplane hashing; later layers use XNOR and bitcount operations for matrix multiplication.Source: §3.3.1 · p. 9
3.3.2 End-to-End Differentiable Training.
- Differentiable training uses a straight-through estimator for sign and an alternative lookup formulation for gradient propagation, while inference uses compact binary representations.Classification and regression use different output treatments and losses.Source: §3.3.2 · p. 9
3.3.3 Predictive System Control Tasks in LeanStream.
- LeanStream predicts neuron activations to prioritize weight loading and computation, and predicts cache-element reuse distance to guide eviction.A PCA-based projection reduces activation-prediction input dimensions by 70%.Source: §3.3.3 · p. 9
4.1 Experimental Setup
LeanStream is evaluated on embedded Jetson platforms and a OnePlus 13, across three 7B models and representative generation tasks. The setup compares it with systems that predict and load weights using earlier-layer information.
Models and Devices
- Experiments use Mistral-7B, Llama2-7B, and Qwen2.5-7B on Jetson AGX Orin, Jetson AGX Xavier, and OnePlus 13.The Jetson devices use Samsung 980 Pro SSDs; the OnePlus 13 uses Snapdragon 8 Elite and UFS 4.0 storage.Source: §4.1 · p. 9
Baselines
- The baselines include DejaVu, which predicts current-layer weights from the previous layer, and PowerInfer-2, which combines weight prediction with in-memory caching.The provided setup passage also names DejaVu+ as an enhanced DejaVu variant with an LRU cache module.Source: §4.1 · p. 9, Figure 9 · p. 10
Datasets
- The evaluation covers long-document question answering with Scrolls-Qasper, factual generation with TruthfulQA, and conversational question answering with CoQA.These tasks span different context lengths and output characteristics.Source: Figure 9 · p. 10
4.2 End-to-end Results
LeanStream outperforms the baselines across tested memory budgets and platforms, with throughput advantages that grow as memory budgets decrease. Its performance varies across model–dataset combinations in memory-bound settings, while accuracy remains close to dense models and baselines.
Overall Performance
- Up to 2.2× higher throughput than PowerInfer-2 at its optimal 50% weight-cache setting is reported for Scrolls-Qasper.LeanStream also outperforms all baselines in each tested scenario, with a larger gap as the memory budget decreases.Source: §4.2 · p. 10
Sparsity and Throughput
- Up to 5.9× throughput is reported while matching PowerInfer-2's peak throughput at its optimal 50% cache setting.The cited figure concerns Mistral-7B on Qasper under varying sparsity levels.Source: Figure 10 · p. 10
Platform Characteristics
- OnePlus 13 becomes I/O-bound earlier as memory budgets shrink, while Jetson AGX Xavier has lower peak throughput than Orin because of its lower compute power.The passage attributes Xavier's strong I/O capability to its SSD matching Orin's.Source: §4.2 · p. 10
Cross-Dataset Analysis
- In memory-bound settings on Jetson AGX Orin, throughput varies significantly across model–dataset combinations because prioritization effectiveness and cache-miss ratios differ.The passage says these results represent similar performance trends on other devices.Source: §4.2 · p. 11
Sparsity and Accuracy Analysis
- LeanStream maintains accuracy nearly identical to the original dense models and the baselines across target weight-sparsity levels.As sparsity decreases, throughput declines because more computation and weight loading are required.Source: §4.2 · p. 11, Table 2 · p. 11
Sparsity and Memory Budgets
- LeanStream maintains a performance lead across tested sparsity levels and remains functional at extremely small memory budgets, where other baselines fail with out-of-memory errors.The supplied figure passage identifies an ablation comparison on Qasper with a 20% weight-caching budget.Source: Figure 11 · p. 11
4.3 Component Impact Study
The component study examines synchronization overhead, prioritization components, cache behavior, and streaming control. Its results highlight prioritized caching and dynamic control as key contributors to performance.
4.3.1 Analysis of Synchronization Overhead.
- Even at 16 splits, LeanStream's synchronization overhead is less than half the original computation time, while traditional synchronization overhead exceeds total computation time.The comparison partitions a Mistral-7B MLP layer on Jetson AGX Orin and OnePlus 13.Source: §4.3.1 · p. 11
4.3.2 Ablation Study of Prioritization Components.
- Prioritized caching provides the largest performance gain among the added components, especially on mobile, where I/O bandwidth is more limited.The study progressively adds prioritized computation, loading, and caching to standard DejaVu; decoding is described as bottlenecked more by weight fetching than arithmetic.Source: §4.3.2 · pp. 11–12
- At a 25% memory budget, LeanStream's cache-miss ratio is 0.11, versus 0.56 for PowerInfer-2 and 0.81 for LRU.At 50% memory, LeanStream's miss ratio is 0.05, described as nearly a fourfold improvement over PowerInfer-2.Source: §4.3.3 · p. 12, Table 4 · p. 12
4.3.3 Effectiveness of Cache Policies.
- LeanStream reduces the volume of weights that must be loaded, which the passage connects to higher end-to-end throughput.This finding is reported in the cache-policy discussion.Source: §4.3.3 · p. 12
- LeanStream's dynamic streaming control consistently outperforms the offline-searched Best Static configuration on Qasper with 20% weight caching on Jetson AGX Orin.The comparison also includes a One Shot configuration that predicts weights once at layer input.Source: §4.3.4 · p. 12, Table 5 · p. 12
4.3.4 Effectiveness of Dynamic Streaming Control.
LeanStream combines a compact stacked-hashing predictor with progressive refinement and streaming control. The evaluation also finds decoding remains the dominant latency contributor, while sustained operation uses more energy per token and reaches higher peak temperatures than the baselines.
4.3.4 Effectiveness of Dynamic Streaming Control.
- The stacked learnable hashing predictor uses approximately 23–24 MB across tested models and predicts in under 100 μs, with 12%–14% relative loading redundancy.The passage contrasts its footprint with 1.1–1.8 GB for DNN predictors and 106–130 MB for BNN implementations.Source: §4.3.5 · p. 12
4.3.5 Efficiency of Stacked Learnable Hashing.
- Progressive refinement partitions each sparse MLP into eight priority-ordered stages and updates the next-layer prediction as stages execute.The prediction refinement is evaluated for Mistral-7B on CoQA under 80% activation sparsity.Source: §4.3.6 · p. 12, Figure 12 · p. 13
4.3.6 Validation across Refinement Steps.
- Importance Ratio and Top-10% Recall improve with progressive refinement; priority ordering yields the largest early-stage gain as high-importance neurons are computed first.The gap between priority and non-priority ordering narrows as both approach the complete sparse MLP output; the trend appears across five representative layers.Source: Figure 13 · p. 13, Figure 12 · p. 13
4.3.7 Analysis of Impact of Prefill Phase.
- Even with a 256-token prompt and 256 generated tokens, decoding remains the dominant contributor to total inference latency.Prefill overhead is mainly attributed to initial dense weight loading and computation; the passage describes prefill as having a higher compute-to-I/O ratio than decoding.Source: §4.3.7 · p. 13
- LeanStream uses slightly more energy per token and reaches higher peak CPU and GPU temperatures than PowerInfer-2 and DejaVu, but does not thermally throttle during two hours of execution.On Jetson AGX Orin, its peak GPU temperature is 76°C, below the 99°C thermal-throttling threshold; CPU and GPU frequencies were fixed for the comparison.Source: §4.3.8 · p. 13, Table 7 · p. 13
5 Conclusion
LeanStream is a streaming speculate-and-refine framework for on-device LLM inference on resource-constrained mobile and embedded platforms. It progressively refines execution, loading, and cache decisions using partial GPU results.
5 Conclusion
- LeanStream progressively refines execution, loading, and cache decisions from partial GPU results to address the tension between context-aware prediction and efficient computation.The framework targets resource-constrained mobile and embedded platforms.Source: §5 · p. 13
6 Disclaimer
The document is informational and includes contributions from JPMC’s Global Technology Applied Research center, but is not a product of its Research Departments. It disclaims warranties and liability, and says it is not investment advice, financial research, or an offer or recommendation concerning securities or services.
6 Disclaimer
- The document is informational and includes contributions from JPMC’s Global Technology Applied Research center.It is not a product of JPMC or its affiliates’ Research Departments.Source: §6 · p. 13
- JPMC and its affiliates make no express or implied representations or warranties about the information’s completeness, accuracy, or reliability, and accept no liability for its use or related outcomes.Source: §6 · p. 13
- The document does not constitute investment advice, financial research, or a recommendation or offer to buy or sell a security, financial instrument, product, or service.Source: §6 · p. 13