PaperlayerView exact revision on arXiv

Source-linked AI summary

Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleets

Tate Berenbaum, Muthaiah Venkatachalam

arXiv:2608.19147v1cs.DCcs.AIcs.SE

TL;DR

Individual Intel AI PCs cannot hold large LLMs, so this paper distributes pre-compiled model shards across a networked fleet; the resulting system reaches 1.79× monolithic single-user throughput and scales to 70B models.

  • Problem

    Individual AI PCs have limited unified memory, leaving evidence needed on whether idle fleets can serve larger LLMs without cloud dependency.

  • Method

    The system partitions models into stateful OpenVINO layer shards, restores GPU cache fusion through beam_idx graph injection, and interleaves speculative, micro-batched requests.

  • Results

    1.79× the monolithic single-user baseline is achieved by two-node 8B inference, while four-node 70B deployment reaches 6.43 tok/s with bit-exact output.

  • Takeaways & Limitations

    A commodity Intel AI PC fleet can provide interactive multi-user inference and distribute models that no single fleet member can host.

  • Takeaways & Limitations

    The system assumes a trusted, reliable network and identical software stacks, and its reported benchmarks are scoped to integrated GPUs.

Abstract

from arXiv · show

Modern Intel AI PCs ship capable integrated GPUs and NPUs with 16+ GB of unified memory, and they spend considerable time idle. That is not enough memory to fit a large model such as a 70B-parameter LLM. We show that a handful of AIPCs, working together over an ordinary network, can serve models beyond the capability of any single one. We use pipeline parallelism: a model is split by layer into per-stage shards, each pre-compiled into an OpenVINO graph, so that every machine runs one shard and passes activations to the next. Three techniques make this fast enough to be useful. First, we recover the speed of the unsplit model: a naive per-stage export runs well below monolithic inference because it misses an OpenVINO GPU optimization, and injecting a beam_idx Gather into each shard triggers that optimization (the IndirectKVCache fusion) and brings the shards to parity. Second, we leverage speculative decoding on stateful OpenVINO models. Third, the pipeline serves several users at once by interleaving their requests across the stages, each request carrying its own cache (micro-batching). Together, a two-node Llama 3.1 8B INT4 pipeline serves two concurrent users at 1.79x the single-user throughput of the unsplit model on the same hardware, and the gap widens under simulated wide-area latency. The same design scales to a 70B model that no single fleet member can hold: a four-node deployment of Lunar Lake AI PCs on Intel Tiber Cloud serves a single user at interactive speed, with output token-for-token identical to the same four-node pipeline decoding without speculation. Code, raw benchmark logs, and reproduction scripts ship as a self-contained package at https://github.com/labscommunity/pipeline-sharded-inference-paper (in the top-level reproduction/ directory).

1 Introduction

The paper presents a distributed inference stack that uses pre-compiled OpenVINO pipeline shards, speculative decoding, and micro-batching to make fleets of Intel AI PCs serve models efficiently over networks. On two consumer PCs, it reaches 1.79× single-user monolithic throughput for two concurrent users and remains usable under simulated WAN latency.

  • Motivation: Intel AI PCs combine integrated GPUs, NPUs, and 16+ GB unified memory, but individual systems serve Llama 3.1 8B at only a few tens of tokens per second.A coordinated fleet could provide cloud-independent inference with full data locality and zero marginal cost.
  • Problem: Standard export tools fail on transformer attention, while monolithic compiled models cannot be split at layer boundaries without invasive graph surgery.Dynamic control flow in rotary embeddings and KV-cache management creates the software barrier.
  • Problem: Distributed autoregressive inference is hindered by per-token network round trips, while canonical speculative decoding assumes paged attention or dedicated rewind APIs.Pipeline parallelism has seen limited evaluation on consumer hardware when stages communicate over real networks.
  • Contributions: The system combines per-stage export, mask-based KV-cache rewind, and pipeline micro-batching to address splitting, speculative decoding, and network-latency bottlenecks.The three techniques are presented as the paper’s contributions.
  • Results: 1.79× throughput is achieved by a two-node consumer Intel AI PC fleet serving two concurrent users versus a single-user monolithic baseline on the same hardware.Under a 100 ms/hop simulated WAN, the configuration stays usable while naïve pipeline-parallel decoding falls below the interactive floor.
  • Contributions: 1.80× system-throughput scaling comes from interleaving independent stateful InferRequests, each carrying its own KV cache, across pipeline stages.The v5_beam Llama 8B shards isolate streams through per-stream compile_model.

2 Background and Related Work

This section positions the work as pipeline inference for Intel AI PCs, where autoregressive decoding’s sequential dependencies distinguish inference from training. It contributes pre-compiled, independently optimized OpenVINO shards, tracing-based export, and stateful speculative decoding while comparing against a same-hardware monolithic baseline.

  • Pipeline parallelism: Pipeline parallelism partitions model layers across devices, but autoregressive decoding imposes strict sequential dependencies because each token depends on the previous one.GPipe and PipeDream established pipeline parallelism for training, where micro-batches keep stages busy.
  • Novelty: The work pre-compiles OpenVINO IR shards and applies beam_idx-Gather graph surgery to unlock IndirectKVCache fusion, achieving monolithic parity per stage.Unlike runtime partitioning, this approach avoids graph overhead and targets Intel integrated GPUs through OpenVINO.
  • Evaluation scope: The evaluation compares against the monolithic openvino_genai.LLMPipeline on identical hardware and reports absolute throughput because other systems use non-overlapping hardware targets.Petals and Parallax use NVIDIA GPUs, MDI-LLM uses Jetson boards, and the proposed system uses Intel iGPUs via OpenVINO.
  • Export pipeline: torch.jit.trace with real tensors and externally precomputed rotary embeddings avoids abstract shape-propagation failures while producing independently optimized per-stage OpenVINO graphs.These failures affect reshape and transpose operations inside modern rotary position embeddings.
  • Micro-batching: Each OpenVINO InferRequest carries independent KV-cache state through ReadValue/Assign operations, enabling temporal request interleaving without framework or additional memory-management changes.The runtime also supports continuous batching, although the results tables do not use it.

3 System Design

The system partitions a HuggingFace transformer into compiled INT4 OpenVINO shards connected by a coordinator-driven TCP pipeline, with stateful KV caching and optional speculative decoding and micro-batching. Its three operating scenarios progressively compose pipeline parallelism, speculative decoding, and concurrent-user execution across the same stages.

  • System architecture: Four components form the runtime: a per-stage export pipeline, TCP activation relay, shard workers, and a coordinator driving autoregressive generation.The coordinator also hosts the tokenizer, generation loop, stage-0 shard, and optional draft model on a fleet node.
  • Activation transport: Each decode step runs stage 0 locally, forwards hidden states through adjacent workers over persistent TCP connections, and returns the final token hop-by-hop.KV-cached decode sends [1, 1, 4096] float32 tensors, or 16 KB per hop, using a 20-byte header and TCP_NODELAY.
  • Shard export: The export pipeline creates N standalone INT4 OpenVINO IR shards covering contiguous layer ranges, with stateful KV cache and 128-element symmetric weight-compression groups.Attention is rewritten with explicit KV tensors and precomputed rotary tensors; numerical equivalence is verified with maximum difference < 5 × 10−7 against HuggingFace’s native forward.
  • KV-cache execution: Stateful ReadValue/Assign operations support one-pass prompt prefill and one-token decode steps, while the coordinator slices precomputed rotary tensors for each position.Models with cross-layer KV sharing transmit shared tensors as additional stage inputs and outputs.
  • Operating scenarios: The runtime exposes PP, PP+SD, and PP+SD+MB: pipeline-only decoding, coordinator-local speculative verification of K+1 positions, and interleaved multi-user streams with independent KV state.The export pipeline supports all scenarios; mask-based KV rewind enables speculative decoding, and micro-batching enters only in PP+SD+MB.
  • Decoding semantics: Beam search is never run; beam_idx is solely a compile-time pattern for OpenVINO cache-reordering fusion, while speculative rejection rewind uses attention_mask.Actual beam search with speculative decoding would require per-beam draft state and candidate-tree verification, which the system does not implement.

4 Reaching Monolithic Parity: the beam_idx Gather Injection

Injecting a beam_idx Gather into each per-stage OpenVINO shard activates the IndirectKVCache optimization and restores near-monolithic throughput without changing model weights or decoding. Splitting still incurs a structural single-machine cost because smaller per-stage workloads reduce GPU occupancy and activation passing adds overhead.

  • 13–23% slower than openvino_genai.LLMPipeline, naive one-stage exports miss the OpenVINO GPU plugin’s IndirectKVCache transformation.The transformation rewrites ReadValue → Concat → Assign patterns into a fused cache operation.
  • A post-export beam_idx: [-1] i32 Parameter and Gather on every KV ReadValue reproduces optimum-intel’s fuse_cache_reorder pass for per-stage exports.The resulting shards are called v5_beam, and the injection occurs at compile time without modifying weights or the decode loop.
  • 15% of throughput recovered, rising from 21.26 to 24.45 tok/s, while v5_beam comes within 0.4% of the monolithic baseline.The same comparison reports roughly 6% Python-loop overhead and v5_beam 5.7% faster than A′.
  • 60.1% mono versus 43.6% on 3-stage, as smaller per-stage GEMMs reduce XVE occupancy and Python activation casting adds cost.Per-stage splitting costs approximately 11–15% relative to Aspec = 24.28 on one machine.

5 Speculative Decoding via Mask-Based KV Rewind

Mask-based KV rewind avoids costly physical cache trimming by masking rejected draft positions, while remaining bit-exact and imposing negligible CPU overhead. Speculative decoding improves throughput across content and generation lengths and composes with sharded inference, but results are scoped to greedy decoding.

  • Physical KV trim: Physical KV trimming costs ∼48 ms per K =3 step, making speculative decoding fall below break-even on Arc B390.The measured trim cost reflects device-side state invalidation rather than NumPy work.
  • Mask-based rewind: Masking rejected cache positions matches physical trimming exactly, with maximum post-correction logit difference 0.0000.The comparison used the same target–prompt–draft–correction sequence and achieved fp32 equality to machine precision.
  • Mask-based rewind: Mask reconstruction costs < 1% of per-step wall time, while K =3 cache bloat reaches 1.02× logical length after 2048-token generation.Periodic compaction is unnecessary for realistic single-turn lengths but remains needed near the context window in multi-turn conversations.
  • Performance: Speculative decoding speedup ranges from 1.11× for creative writing to 1.50× for code completion, tracking draft acceptance rates from 49.7% to 93.1%.Across 128- to 2048-token runs, acceptance rises from 70.7% to 97.5%, while K =3 speedup increases from ∼1.3× to ∼1.6×.
  • Sharded inference: The multiplier is 1.24× on the monolithic target and 1.32× on the 3-stage shard target, with bit-exact outputs on both paths.The wrapper-level MaskedReq abstraction composes with monolithic and sharded OpenVINO targets; reported results use greedy decoding.

6 Distributed Pipeline Evaluation … 6.3 Per-Token Breakdown

The distributed evaluation uses a three-AIPC OpenVINO GPU testbed and progressively combines micro-batching with speculative decoding. The resulting two-node pipeline reaches 43.97 tok/s, while per-token analysis attributes about 10% of time to network round trips and reports TTFT up to 786 ms.

  • 6.1 Testbed: Three Intel AI PCs—two Lunar Lake Zenbook S 14 nodes and one Panther Lake OmniBook X 16—run GPU inference over the same 802.11ax WiFi network.Each machine has 32 GB of memory; the nodes use Arc 140V or Arc B390 integrated GPUs.
  • 6.1 Testbed: The testbed uses Windows 11, Python 3.11, OpenVINO 2026.1.0, and GPU inference, with 6.5 ms raw TCP round trips for 16 KB payloads.The TCP measurement was taken separately on the 802.11ax LAN.
  • 6.2 Progressive Optimization: 16.33 tok/s is the starting throughput for the 2-node v5_beam pipeline, versus 24.54 tok/s for the single-node monolithic reference.The 20 ms per-token gap comprises approximately 6 ms of TCP round-trip time and 14 ms of Python-wrapper plus OpenVINO dispatch overhead.
  • 6.2 Progressive Optimization: Micro-batching provides a 1.80× multiplier, increasing distributed throughput from 16.33 to 29.34 tok/s by interleaving two independent requests across pipeline stages.Each shard uses two InferRequests, allowing stage-0 computation for one request while stage 1 processes another.
  • 6.2 Progressive Optimization: 1.50× speculative decoding on top of micro-batching yields 43.97 tok/s, or 1.79× the monolithic single-user reference while serving two users concurrently.The K+1-token target verification batches four tokens per TCP round trip.
  • 6.2 Progressive Optimization: The measured composition is empirical: 16.33×1.797×1.499 = 43.99, matching 43.97 within 0.05 tok/s.All three multipliers were measured in the same paired session.
  • 6.3 Per-Token Breakdown: TTFT on the 2-stage distributed pipeline ranges from 122 ms for an 8-token prompt to 786 ms for a long multi-pass prefill.The variation is dominated by prompt-length-dependent computation on stage 0 rather than network overhead.
  • 6.3 Per-Token Breakdown: Dedicated TCP benchmarking measured approximately 6 ms per round trip, about 10% of per-token time, correcting an initial “70% network time” metric that included remote-worker computation.The remainder is compute and Python overhead, including OpenVINO C++ dispatch and state management.

6.4 Activation Compression · 6.5 WAN Latency Sensitivity

Activation compression does not improve LAN performance because ∼6.5 ms per-hop cost is dominated by latency, while WAN conditions make transport and message amortization more important. Under simulated WAN latency, the full stack increasingly outperforms naïve distributed decoding, with speculative decoding and micro-batching preserving interactive throughput through 100 ms/hop.

  • 6.4 Activation Compression: Under gigabit WiFi, 16 KB transmits in under 0.2 ms, while the ∼6.5 ms per-hop cost comes from TCP scheduling and round-trip time.Halving the activation payload therefore gains nothing on LAN.
  • 6.4 Activation Compression: INT8 symmetric quantization of the 4096-dim hidden state corrupts generation, so activation compression is relevant only under bandwidth-constrained WAN conditions.Per-channel or group quantization might preserve quality but was not tested.
  • 6.4 Activation Compression: QUIC could remove TCP head-of-line blocking, but LAN-decode gains should be modest, UDP may be blocked, and a production transport still needs TCP fallback.QUIC is described as the most promising future work in this layer, but it has not yet been benchmarked.
  • 6.4 Activation Compression: On LAN, wired gigabit Ethernet makes the same 16 KB round trip sub-millisecond, removing ∼10% of the per-token budget.Speculative decoding and top-1 logits compression already reduce network overhead, leaving the network at ∼10% of per-token LAN time.
  • 6.5 WAN Latency Sensitivity: The full-stack-over-naïve multiplier grows from 2.83× at LAN to 4.04× at 100 ms/hop.Micro-batching folds two streams into one pipeline, while speculative decoding amortizes per-hop latency through K +1-token verification.
  • 6.5 WAN Latency Sensitivity: At 100 ms/hop, the full stack remains usable while naïve distributed decoding falls below the interactive floor past ∼25 ms/hop.The latency sweep injects one-way delay at worker recv() and send() boundaries, with reported speedups considered lower bounds because sleep-based emulation serializes messages.
  • 6.5 WAN Latency Sensitivity: At 100 ms/hop, K = 10 reaches 11.30 tok/s, or 4.07× over naïve 2.77, because packing more tokens per round trip dominates under network-bound conditions.At 50 ms/hop, K =10 reaches 16.04 tok/s, or 3.36× over naïve 4.77.
  • 6.5 WAN Latency Sensitivity: K = 7 peaks at 30.96 tok/s on LAN, or 1.44× over the single-stream LAN baseline of 21.55 tok/s.Moderate K values balance acceptance with draft cost when compute dominates.

6.6 3-Stage Full Stack and 3-Stream Concurrency

The full-stack 3-stage pipeline improves throughput over the 2-stage baseline while preserving bit-exact outputs across concurrent streams. Under simulated wide-area latency, increasing speculation K enables the 3-stage stack to outperform the 2-stage result by amortizing additional network hops.

  • 3-Stream Concurrency: All configurations are bit-exact across both or all streams.The third stage adds compute parallelism for micro-batching, keeping three iGPUs busy per stream.
  • 3-Stage Full Stack: 50.55 tok/s: the 3-stage 2-stream stack exceeds the 2-stage K =3 headline of 43.97 tok/s by 15%.This uses K =5 on the Llama 3.1 8B INT4 v5_beam configuration.
  • 3-Stream Concurrency: +14.1 tok/s: moving from 2-stream to 3-stream at K =5 raises aggregate throughput from 50.55 to 64.67 tok/s.The increase costs ∼3.7 tok/s per-user latency and reaches the min(Nusers, Nstages) pipeline-fill ceiling.
  • Wide-Area Latency: 14.97 tok/s: at L=100 ms/hop, the 3-stage 2-stream stack with K =10 reaches 1.34× the 2-stage K =3 result of 11.20 tok/s.The higher K amortizes the extra network hops because K+1 tokens are emitted per target round-trip.

6.7 Top-1 Logits Compression · 6.8 Real-WAN Validation on Tiber Cloud · 6.9 Long Generation and Stability

Top-1 logits compression removes the WAN bottleneck caused by segmented logits transfers, delivering interactive throughput over Tiber Cloud’s DERP-relayed path. Extended-generation tests show stable or improving speculative-decoding gains, while prior distributed measurements showed no sustained per-token degradation but are not revalidated on v5_beam.

  • 6.7 Top-1 Logits Compression: Top-1 compression encodes only (arg max, pmax), reconstructs one-hot logits, and preserves bit-exact greedy speculative decoding.The acceptance check compares token IDs, so agreement on arg max is sufficient.
  • 6.7 Top-1 Logits Compression: +12% Pareto improvement on LAN results from avoiding TCP congestion-window costs for the 501 KB logits payload.The activation tensor is only 16 KB, while vocabulary-sized logits are approximately 501 KB and fragment across TCP segments.
  • 6.7 Top-1 Logits Compression: 8.17× improvement over uncompressed logits on the Tiber path makes top-1 compression more impactful than activation compression.The Tiber ratio was re-measured on OV 2026.1.0 and exceeded the prior 5.57× ratio.
  • 6.8 Real-WAN Validation on Tiber Cloud: The Tiber experiment used two Lunar Lake / Arc 140V instances running identical v5_beam shards, with traffic forced through Tailscale DERP relays at approximately 32 ms instance-to-instance RTT.The configuration used Python 3.14 and OpenVINO 2026.1.0; Intel’s network blocked direct UDP between instances.
  • 6.8 Real-WAN Validation on Tiber Cloud: 22.88 tok/s aggregate (11.44 per stream) with top-1 compression recovers interactive throughput, versus 1.40 tok/s for both uncompressed streams.The uncompressed path falls below the 5 tok/s interactive floor, while compression makes the DERP-only topology viable.
  • 6.8 Real-WAN Validation on Tiber Cloud: At comparable WAN conditions, throughput differs across methods: 11.20 tok/s for 2-stage sleep simulation, 2.01 for 3-stage queue proxy, and 2.80 uncompressed versus 22.88 compressed on Tiber.These results support per-segment relay queueing, rather than one-way latency alone, as the dominant real-WAN constraint.
  • 6.9 Long Generation and Stability: Speculative-decoding speedup rises from 1.33× at 128 tokens to 1.58× at 2,048 tokens as acceptance increases from 70.7% to 97.5%.At 512 tokens the speedup is 1.56× with 90.6% acceptance; at 1,024 tokens it is 1.57× with 95.1% acceptance.
  • 6.9 Long Generation and Stability: Prior 2-stage distributed measurements showed 15.95 tok/s at 200 tokens, 15.55 tok/s at 1,000 tokens, and 14.46 tok/s aggregate across 10 prompts, but predate v5_beam.The workload used a 36 ms KV-cache reset between prompts, and sustained long-generation performance on the full stack was not re-measured.

6.10 Gemma 4 E2B: A Second Architecture · 6.11 Llama 3.1 70B: 4-Stage Distributed on Tiber Cloud

Gemma 4 E2B required architecture-specific export and cache handling, while the four-stage Llama 3.1 70B deployment demonstrated distributed inference beyond single-node memory capacity with speculative decoding over Tiber Cloud.

  • 6.10 Gemma 4 E2B: A Second Architecture: Gemma 4 E2B uses FP32 across 35 layers because of PLE quantization sensitivity, with shared K/V projections requiring cross-stage transmission of L13/L14 cache tensors.The shared-K/V layers read source-layer caches through DynamicCache, so stage 0 emits additional non-stateful outputs.
  • 6.10 Gemma 4 E2B: A Second Architecture: Rotary-fixed Gemma shards recover full speed under OpenVINO 2026.1, avoiding the INFERENCE_PRECISION_HINT="f32" workaround that costs −10% on two stages and −60% on one.OpenVINO 2026.1 strictly rejects the mixed-precision rotary graph produced by tracing, motivating re-export with a rotary fix.
  • 6.10 Gemma 4 E2B: A Second Architecture: 13.35 tok/s makes v2_beam 4.5% faster than v2 at 12.78 tok/s in the two-stage in-process Gemma result.In distributed mode, network transfers of hidden states and cross-KV tensors dominate, preventing the one-stage compute advantage from carrying over.
  • 6.10 Gemma 4 E2B: A Second Architecture: 16.30 tok/s is the two-stream Gemma v2 aggregate throughput, representing 1.57× single-stream throughput with byte-identical concurrent outputs.The strong 10.40 tok/s single-stream baseline limits the micro-batching ratio because less stage-idle time is available to fill.
  • 6.11 Llama 3.1 70B: 4-Stage Distributed on Tiber Cloud: Llama 3.1 70B was exported as four v5_beam INT4 shards, each about 9 GB, totaling about 36 GB across four Tiber Cloud AI PC instances.The deployment uses 20 layers per shard, embedding on stage 0, lm_head on stage 3, and Tailscale DERP communication.
  • 6.11 Llama 3.1 70B: 4-Stage Distributed on Tiber Cloud: 5.42 tok/s at K=10 is a 3.1× speedup over the same-topology target-only baseline of 1.74 tok/s, with bit-exact output for the first 10 tokens.The single-stream peak results from amortizing three remote worker round-trips across approximately 7.5 emitted tokens per speculative call.
  • 6.11 Llama 3.1 70B: 4-Stage Distributed on Tiber Cloud: 5.72 tok/s at 1024-token context is 5.5% above the 128-token reference, with acceptance reaching 72.2%; at 4096 tokens, throughput falls to 5.00 tok/s and acceptance to 66.3%.The initial improvement reflects higher draft-target agreement after context establishment, before larger KV caches reverse the trend.
  • 6.11 Llama 3.1 70B: 4-Stage Distributed on Tiber Cloud: Lunar Lake Arc 140V handled 20-layer 70B INT4 stages cleanly, whereas an Arrow Lake-S Xe-LPG deployment lost an iGPU during inference and monolithic export was OOM-killed.The monolithic attempt exceeded available memory when FP16 calibration weights and NNCF compression workspace were buffered; bit-exact four-stage equivalence remains the strongest correctness check.

6.12 Scenario Comparison: PP vs. PP+SD vs. PP+SD+MB

The comparison identifies speculative decoding as the preferred single-user mode and micro-batching as the preferred multi-user mode, while PP-only remains a fallback when draft-model resources or compatibility are unavailable. Operating choices depend on latency, user count, model size, and interactive-throughput requirements.

  • Deployment implications: Speculative decoding is bit-exact with greedy decoding and is faster than PP-only, with measured multipliers of 1.24–1.44× in-process and 3.11× over a 4-hop WAN relay.PP-only is mainly justified when coordinator iGPU memory for the draft is unavailable or no compatible draft model exists.
  • Deployment implications: Scenario (b) maximizes per-user tokens/s and uses K =5–7 on LAN or K =10 at ≥50 ms/hop.It is identified as the single-user operating point, with K selected according to network latency.
  • Deployment implications: Scenario (c) maximizes aggregate throughput for multiple users, constrained by pipeline fill and per-stream compile_model memory of ∼6 GB iGPU per Lunar Lake stream.Its per-user cost increases as aggregate throughput is optimized.
  • Deployment implications: For 8B on LAN, two micro-batched streams leave each user at ∼4× the interactive floor, making scenario (c) the clear serving default.This is the multi-user operating point described for 8B LAN deployment.
  • Deployment implications: For 70B over WAN, a second stream lowers per-user throughput below 5 tok/s, so scenario (c) suits batch serving while an interactive user should remain on (b).The recommendation differs by workload: multi-user batch serving versus interactive single-user service.

7 Multi-User Throughput via Micro-Batching

Micro-batching interleaves independently cached user requests across pipeline stages to fill idle bubbles and improve utilization. The v5_beam Llama pipeline reaches 1.80× two-stream scaling, while three streams add 14.1 tok/s aggregate on a three-stage testbed.

  • Motivation: Balanced 25 ms stages provide roughly 50% utilization for a single request because one stage idles while the other computes.This inefficiency motivates interleaving multiple requests across stages.
  • Method: Independent InferRequests and compiled graphs give each stream its own KV cache state, enabling alternating requests to fill pipeline bubbles.The implementation costs seconds of extra compile time per Arc iGPU stream and ∼2 GB of GPU memory per additional Llama 8B stage per stream.
  • Two-stream results: 1.80× is the measured v5_beam two-stream scaling, below the theoretical 2× maximum because the faster 16.33 single-stream baseline leaves less idle time to absorb.The original measurement window reached 2.03× with a 14.51 baseline.
  • Beyond two streams: 14.1 tok/s aggregate is added when three concurrent streams replace two on the three-stage testbed, rising from 50.55 to 64.67 tok/s and yielding 1.28× scaling.Per-stream throughput falls from 25.3 to 21.6 tok/s, remaining 4.3× the 5 tok/s interactive floor.

8 Discussion

The discussion shows that distributed Intel AI PC fleets can serve larger models with interactive, private inference, including a bit-exact 70B deployment, while identifying constraints in latency tuning, model precision, caching granularity, and NPU performance. It also presents practical serving mechanisms for prefix reuse and concurrent-request isolation.

  • Fleet tradeoffs: 43.97 tok/s aggregate serves two concurrent Llama 3.1 8B users at zero marginal cost and full data locality.At 100 ms/hop WAN latency, speculative decoding remains interactive at 11 tok/s aggregate, versus 2.77 tok/s for a naïve pipeline.
  • 70B validation: 5.42 tok/s on a 4-node Llama 3.1 70B INT4 fleet exceeds the 5 tok/s interactive floor and is bit-exact against target-only decoding.The same measurement reports 5.95 tok/s with two streams and 6.43 tok/s with a Panther Lake coordinator; speculative decoding yields a 3.1× speedup over 1.74 tok/s.
  • Latency tuning: K =10 is optimal at 100 ms/hop and amortizes approximately 7.5 emitted tokens per target round-trip in the 4-hop 70B topology.At K =10, each emitted token pays approximately 13% of the round-trip latency of the naïve K =1 path.
  • Latency tuning: At 1024 tokens, acceptance rises to 72.2% and throughput reaches 5.72 tok/s as speculative decoding benefits from prompt-provided context.This reverses the usual per-step compute trend described in the passage.
  • Limitations: Gemma 4’s 262K × 8960 Per-Layer Embeddings require FP32 because INT4 and INT8 produce garbage, inflating stage 0 to 7 GB beyond Lunar Lake’s approximately 4 GB GPU allocation.A mixed-precision export path could address this, but NNCF would need per-subgraph precision control.
  • Caching limitations: Captured prefix states enable whole-sequence session resume, but opaque shard-local state prevents block-level sharing of partial prefixes.Spec-decoded sessions are warm-resumable after realigning the draft model before feeding the shared suffix.

9 Conclusion

Commodity Intel AI PC fleets can provide interactive, multi-user LLM inference beyond single-machine capacity, including across links where naïve pipeline parallelism fails. Three independently validated techniques address shard optimization, speculative decoding overhead, and stage-idle time.

  • Fleet-scale inference: 6.43 tok/s 2-stream for four-node Llama 3.1 70B INT4 over a Tailscale DERP relay, 3.1× over the same-topology target-only baseline, bit-exact.This demonstrates distribution beyond 8B to models requiring multiple machines.
  • Validated techniques: The three composing techniques each address a distinct bottleneck and are individually validated with bit-exact output against their respective baselines.The techniques target per-stage export performance, stateful speculative decoding, and pipeline utilization.
  • Validated techniques: Beam_idx Gather injection unlocks IndirectKVCache fusion for per-stage exports, producing shards at monolithic parity.The fusion is provided by the OpenVINO GPU plugin.
  • Validated techniques: Mask-based KV-cache rewind avoids the ∼48 ms-per-call query_state/set_state round-trip that would otherwise make speculative decoding a net loss on stateful OpenVINO.The technique addresses speculative decoding overhead in stateful OpenVINO models.
  • Validated techniques: Independent-state micro-batching fills stage-idle windows, isolating each stream through its own compile_model and yielding 1.80× scaling at the 2-stream operating point.The idle windows become smaller after v5_beam fusion.
  • Reproducibility: Code, export scripts, raw benchmark logs, and VTune reports are available in the repository’s top-level reproduction/ directory.The package includes the export pipeline, distributed coordinator and worker, benchmark drivers, and per-section reproduction scripts.
Loading 2608.19147v1…