Source-linked AI summary
Cascadia: A Control-Plane-Free Alternative to Hyperconverged AI Infrastructure
Matias Parij, Pawan Paudel, Tate Berenbaum, Muthaiah Venkatachalam
TL;DR
Client-device fleets offer heterogeneous, intermittently available resources, while established serving systems assume dedicated clusters. Cascadia embeds serving across those nodes and reports higher response throughput in tested multi-node deployments than the stated single-node baselines.
Problem
Client-device fleets have heterogeneous, intermittently available resources, and established enterprise serving stacks assume dedicated clusters rather than these conditions.
Source: §1 · p. 1Method
Cascadia embeds ingress, scheduling, and inference in fleet nodes, with an operator-run CA handling admission and fleet management outside the inference path.
Source: §3.1 · p. 5Results
3.10× response throughput at three nodes relative to one was reported in the June study, and a separate four-node deployment recorded 4.06× throughput over direct single-node serving.
Source: §1 · p. 2, §10 · p. 20Takeaways & Limitations
The results support client fleets as a practical serving substrate with a different deployment footprint from managed datacenter clusters.
Source: §10 · p. 20The system assumes a single-operator, permissioned mesh, and its direct-stream state plane alone does not establish large-fleet scalability.
Source: §9 · p. 19
Abstract
from arXiv · showhide
We present Cascadia, a system for serving large language models on fleets of commodity Intel AIPCs using their CPU, integrated-GPU, and NPU resources. Every node embeds ingress, scheduling, and execution; inference requests require no dedicated routing control plane. Nodes join a libp2p QUIC mesh using CA-issued ed25519 admission certificates, gossip signed capabilities, exchange live load over direct peer streams, and route OpenAI-compatible requests to eligible peers. An operator-run certificate authority handles admission and fleet management outside the inference path. Three serving modes share one interface: whole-model execution on one node, load-balanced replicas, and pipeline-sharded chains using the compilation and speculative decoding mechanism of our companion paper. Optional KV-cache mobility reuses compatible conversation prefixes after a routing move, with cold recomputation on a miss. Signed response receipts and hash-chained logs support provenance and audit. A three-node Phi-3.5-mini NPU testbed delivered 3.10x the response throughput of its one-node configuration under ten concurrent requests; a separate four-node deployment recorded 4.06x the throughput of direct single-node serving. Paired latency observations, runtime measurements, and internal functional checks characterize the tested configurations. We compare Cascadia with IBM, Nutanix, VMware, and HPE platforms on deployment footprint, hardware requirements, scheduling, scaling, licensing, and trust, using vendor documentation. The paper repository provides benchmark scripts, curated measurements, and a claim-to-evidence map.
1 Introduction
Client-device fleets offer substantial but heterogeneous and intermittently available resources, while established enterprise serving stacks assume dedicated clusters. Cascadia operationalizes fleet serving through per-node request handling, multiple serving modes, and cryptographic trust, then reports measurements and a vendor-documentation comparison.
Motivation
- Established enterprise stacks assume dedicated clusters, while client fleets are heterogeneous, intermittently available, and may lack a node large enough for a target model.The paper frames its question as how to turn such fleets into dependable LLM serving systems.Source: §1 · pp. 1–2
System contribution
- Cascadia embeds ingress, scheduling, certificate verification, and audit in identical nodes, with certificate issuance and fleet management retained by an operator-run CA.The design uses existing fleet hardware and avoids a dedicated request-routing control plane.Source: §1 · p. 2
Serving modes
- The system unifies single-node execution, replicated whole-model serving, and pipeline-sharded chains behind one OpenAI-compatible surface.Session affinity and optional KV transfer manage reusable state when routing changes.Source: §1 · p. 2
Trust
- Its trust layer combines CA-issued admission certificates, signed response receipts, and hash-chained logs for independently checkable provenance and audit.The stated revocation measurement is 18–20 s against a ≤60 s target.Source: §1 · p. 2
Evaluation
- 3.10× response throughput at three nodes relative to one was reported in the June study; a separate four-node deployment reached 4.06× throughput over direct single-node serving.The evaluation distinguishes performance measurements from functional validation and identifies each measurement session.Source: §1 · pp. 2–3
Comparison and scope
- The paper compares IBM, Nutanix, VMware, and HPE platforms on deployment and operational requirements using vendor documentation, while noting that it does not model total cost of ownership.Cascadia uses already-owned hardware rather than requiring dedicated inference servers or rented third-party compute.Source: §1 · p. 3
2 Background and Related Work
The background contrasts managed server-cluster platforms with distributed inference on commodity devices and places Cascadia alongside related systems. It also summarizes the companion paper’s sharded-chain mechanism and measured results.
2.1 The Hyperconverged Enterprise AI Paradigm
- The documented enterprise paradigm centers on managed server clusters with dedicated coordination services and provisioned inference hardware.The comparison concerns documented configurations and their hardware and operational requirements; distributed model execution is not unique to Cascadia.Source: §2.1 · pp. 3–4
2.2 Distributed Inference on Commodity Hardware
- Petals, Parallax, MDI-LLM, and exo explore distributed inference on consumer or edge devices, while Cascadia targets Intel client silicon and combines replication with pipeline sharding.Cascadia also distinguishes itself through an enterprise trust layer and no baked-in seed, server, or coordinator role.Source: §2.2 · p. 4
- AMD’s documented client-device example uses hand-configured llama.cpp RPC without discovery or placement, while single-device AI-PC products do not pool machines into one endpoint.The paper says it is not aware of an existing layer pooling Intel client machines into a single served endpoint.Source: §2.2 · p. 4
- Cascadia addresses conversation-state placement across pipeline chains with affinity, compatibility checks, restoration, and cold fallback, evaluated through scoped functional checks.The paper does not import performance claims from disaggregated datacenter systems.Source: §2.2 · pp. 4–5
- The paper benchmarks against direct single-node serving on identical hardware and model, reporting absolute numbers rather than comparing systems with non-overlapping hardware targets.Its sparse-MoE runtime also adopts techniques from prior systems and methods.Source: §2.2 · p. 5
2.3 The Companion Paper: the Sharded-Chain Mechanism
- The companion paper reports mask-based KV-cache rewind, multi-user micro-batching, and a four-stage Llama 3.1 70B INT4 deployment across four Lunar Lake AI PCs.That deployment reached interactive throughput on a model that fits no single member, validated bit-exact against its same-topology baseline; results used the Python research stack.Source: §2.3 · p. 5
3 System Overview
Cascadia embeds gateway, scheduling, discovery, trust, and inference components in identical fleet nodes while keeping CA management outside per-request scheduling. Its design tenets and request lifecycle describe how nodes accept, route, and return authenticated inference responses.
3.1 Two Planes, Three Binaries
- Cascadia ships as three binaries: identical cascadia-node instances, an operator-run CA for admission and fleet management, and a relay for NAT-restricted peers.The node embeds the HTTPS gateway, scheduler, gossip and discovery, trust machinery, and inference backends.Source: §3.1 · p. 5
- Control-plane-free means inference needs no dedicated routing or scheduling service, although admission, revocation publication, and operator-initiated fleet changes still use management services.Any node can accept a request, schedule it, execute locally, or forward it to a peer.Source: §3.1 · p. 5, §3.2 · p. 6
3.2 Design Tenets
- The design makes every node eligible for entry, scheduling, and execution, and grounds trust in certificates, signed advertisements, receipts, and hash-chained logs.Engines remain pluggable behind a byte-blind orchestration layer; Intel CPU, integrated-GPU, and NPU access uses OpenVINO, with device support depending on platform and driver.Source: §3.2 · p. 6
3.3 Request Lifecycle
- A receiving node verifies the bearer token, then selects an eligible peer by capability, load, and peer-health rank, possibly executing locally.The surface is OpenAI-compatible at every node.Source: §3.3 · p. 6, §3.2 · p. 6
- Forwarded requests use authenticated libp2p streams; responses return as chunks and a signed receipt, while failures trigger a typed error or one alternate-peer retry.The entry node rejects responses from unadmitted or revoked peers, and both sides append the exchange to hash-chained event logs.Source: Figure 1 · p. 6, §3.3 · p. 7
- The system versions inference, gossip, state, and pipeline protocols independently, and uses a separate stream for KV mobility.Breaking changes require compatible peers to upgrade.Source: §3.3 · p. 7
4 Serving Modes
Cascadia uses one mesh and scheduler to serve models as single-node executions, replicas, or pipeline-sharded chains. Session affinity and optional KV mobility address state when routing changes.
4.1 Mode 1: Single-Node Whole-Model
- Single-node whole-model serving uses OVMS or embedded engines, with OVMS providing batching and KV-cache management while NPU configurations remain sequential.Embedded engines can add speculative decoding using a draft model or prompt-lookup drafting.Source: §4.1 · p. 7
- 2.03 to 2.91 tok/s (+43%) for Yi-1.5-9B fp16 at 99.8% of nominal iGPU capacity, after moving the embedding to CPU.Comfortably fitting models were best left on iGPU alone, and the NPU was the slowest decode tier in every measured configuration.Source: §4.1 · p. 8
4.2 Mode 2: Replicated Whole-Model
- Replica scheduling combines signed model capabilities with separate live-load streams, local admission reservations, and peer-health tracking.A heterogeneous mesh exposes a union of models through one entry point and routes requests using capability and load.Source: §4.2 · p. 8
4.3 Mode 3: Pipeline-Sharded Chains
- Pipeline-sharded models use ordered worker chains admitted only when stage placement, memory fit, engine kind, and build versions are compatible.The encrypted-mesh checks validated an 8B chain and mixed replica/chain routing; the companion paper’s 70B result was not measured on the encrypted mesh path.Source: §4.3 · pp. 8–9
- 29.62 tok/s at 4096-token generations on a 2-stage Thunderbolt-linked pair using distributed speculative decoding.Overlapping network and draft compute added 6–19%, depending on K.Source: §4.3 · p. 9
4.4 Mode 3, Extreme Case: Sparse-MoE with Disk-Streamed Experts
- 0.105 output tokens per second at K′=8 rose to 0.325 tok/s at K′=4 for Kimi K2.6 on one disk-bound host.This is a capability demonstration, not an interactive-serving claim; the numerical MoE observations concern single-host execution.Source: §4.4 · p. 9
4.5 Mode Selection
- Mode selection favors replicas when a model fits one node, chains when it fits aggregate fleet memory, and disk-streamed experts when it exceeds that memory.Sharding a fitting model costs roughly 11–15% overall in the cited single-stream configuration, while chains can recover and surpass parity at multi-stream operating points.Source: §4.5 · pp. 9–10
4.6 Session Affinity and KV-Cache Mobility
- Session affinity prefers the previous eligible chain, while optional KV pulls validate compatible prefixes and fall back to cold prefill on misses or failures.A separate opt-in mechanism can resume an open response on another chain using emitted token IDs; it does not require the failed chain’s KV to remain reachable, and neither mechanism makes arbitrary engines or versions interchangeable.Source: §4.6 · pp. 10–11
5 Trust, Admission, and Provenance
Cascadia places trust in cryptographic identities and artifacts rather than relying solely on a cluster boundary. Its mechanisms support admission, response verification, local audit, and revocation.
Admission and authentication
- Nodes require CA-signed admission certificates, while partners and operators use distinct PASETO token roles and signing roots.Peers verify admission before inference traffic flows.Source: §5 · p. 11
Response provenance
- Each executing node signs a receipt binding request hash, response hash, and executor identity, which an external verifier can check offline.A receipt is not a hardware-attestation claim and does not prove imported KV was computed correctly.Source: §5 · p. 11
Audit logs
- Signed per-node events form hash-chained append-only logs, with verification detecting any byte flip.Receipts identify the executor and response; the log links each local event to its predecessor.Source: §5 · p. 11
Revocation and replay
- Three internal admission trials measured mesh-wide revocation propagation in 18–20 s against a ≤60 s objective.Entry-node rejection depends on that node learning the revocation; replayed request IDs are rejected worker-side.Source: §5 · p. 11
Trust boundary
- KV tenant isolation assumes admitted workers relay tenant identities honestly and does not protect against a compromised admitted worker.Receipts attest output provenance, not the correctness of imported KV.Source: §5 · p. 12
Hardware constraint
- NPU execution is not bit-deterministic across machines, so response hashes cannot be compared bytewise across real-NPU executors.Receipt verification that an executor produced a response for a request is unaffected.Source: §5 · p. 12
6 Operability and Resilience
Cascadia provides setup, deployment, and network-support mechanisms for AI PC fleets. Its disconnected deployment posture uses static binaries and model files rather than a cluster bootstrap path.
Setup and packaging
- A setup wizard, connectivity preflight, and bootstrap command support installation, dependency setup, and model retrieval.Nodes install as Windows Services or Linux systemd services, and onboarding scripts check serving readiness on Windows and Ubuntu.Source: §6 · p. 12
NAT traversal
- Residential and branch nodes use NAT detection, hole-punching, relay fallback, and rendezvous endpoints to establish peer connectivity.Functional evidence covers simulated full-cone and symmetric NATs, not all network configurations.Source: §6 · p. 12
Failure behavior
- Before streaming, bounded retries and peer-health demotion route around failures; after streaming begins, an exhausted chain returns a typed error rather than silent truncation.Worker death prevents KV pull from that worker, and functional checks do not establish a recovery-time guarantee.Source: §6 · p. 12
Fleet management and observability
- The CA manages deployment and stage-reassignment jobs separately from decentralized request routing, while metrics and traces expose routing and lifecycle activity.A deployment acknowledgement records an assignment for the next start, not proof that a new sharded model is already serving.Source: §6 · p. 13
Air-gap posture
- Disconnected operation is the normal deployment case, using static binaries and model files without an image registry, operator catalog, or cluster bootstrap.The CA runs wherever the operator’s trust boundary is.Source: §6 · p. 13
7 Evaluation
Across dated AI-PC deployments, Cascadia showed fleet-throughput gains; runtime measurements and KV checks characterize specific configurations, while the evaluation distinguishes measured performance from functional validation.
7.1 Method and Testbeds
- The June latency study measured 20 requests per arm at concurrency 1, while its scaling study used 60 requests per fleet size at concurrency 10.May and June deployments were retained separately, and the system description reflects the September 2026 implementation.Source: §7.1 · p. 13, §7 · p. 13
7.2 Observed Latency and Fleet Throughput
- 4.06× aggregate throughput and 6.86× lower median latency under load were recorded for the four-node deployment versus one direct-serving machine.Table 1 describes the historical May deployment; its aggregate and loaded-latency observations used ten concurrent requests.Source: Table 1 · p. 14, §7.2 · p. 14
7.3 Replica Scaling
- 3.10× response rate at three nodes and 2.17× at two nodes were measured relative to the one-node run.The June replica study used 60 requests per fleet size at concurrency 10; a mislabeled run that routed to two workers was excluded.Source: Table 2 · p. 14
7.4 Trust-Layer Latencies
- Trust-layer observations included three internal revocation trials, peer discovery for a fourth node, and receipt-bearing streamed and non-streamed responses.The public receipt session included 20 streamed responses, ten non-streamed responses, and 50 model-list probes; receipt presence does not measure verification cost.Source: §7.4 · p. 14, Table 3 · p. 15
7.5 Runtime-Level Results
- Runtime placement results show ILP benefits only near full iGPU pressure, while NPU-overflow placements were worse in every measured configuration.The placement table reports a four-trial mean for Yi-1.5-9B and single-trial results for other rows.Source: Table 4 · p. 15, §7.5 · p. 15
7.6 KV Mobility: Functional Validation
- KV checks validated compatible-prefix transfer and scoped continuation, but the sparse-MoE fixture was small and a Qwen3 check showed warm/cold divergence.The checks do not establish full-size-model throughput, latency gains over cold recomputation, or durable session service; KV coordination remains default-off with engine and model restrictions.Source: §7.6 · pp. 15–16
7.5 Runtime-Level Results
- 0.156 tok/s at K′=6 versus 0.115 tok/s at K′=8 (+36%) came from a separate session; expert reduction changes computation, so these are not quality-matched speedups.The Kimi K2.6 experiment used one disk-bound host with 133 GB RAM for a model of about 553 GB INT4.Source: Table 5 · p. 16
8 Comparison with Hyperconverged Enterprise AI Infrastructure
The section compares Cascadia with four enterprise platforms using vendor documentation, while noting that their supported hardware configurations do not permit a clean head-to-head throughput comparison. It describes a distinct client-fleet deployment tier and scopes the comparison accordingly.
8.1 Footprint and Control Plane
- All four incumbent stacks require a functioning control plane before serving, while Cascadia uses zero dedicated machines and keeps CA admission and fleet management outside the inference path.A one-node Cascadia deployment runs the same software as larger deployments.Source: §8.1 · p. 16
8.2 Hardware Gating and Capital Expenditure
- Incumbent configurations require suitable server hardware, while Cascadia reuses already-purchased AI PCs; the authors do not claim a single TCO because costs vary by organization.The comparison defines an accelerator floor as the smallest supported managed-inference configuration, not a research possibility.Source: Table 6 · p. 17, §8.2 · p. 18
8.3 Scaling Model
- Both datacenter runtimes and Cascadia support model parallelism, but Cascadia composes advertised stages on admitted client machines and routes requests locally rather than using managed server-cluster environments.The companion paper reports the four-AI-PC 70B runtime result; this paper gives separate functional evidence for mesh integration [5].Source: §8.3 · p. 18
8.4 Licensing Structure
- Incumbents meter by provisioned capacity, including on-prem watsonx at $643,200/year for 67 VPCs; Cascadia’s runtime is Apache-2.0, but its closed-source enterprise mesh terms and total cost are not established.Source: §8.4 · p. 18
8.5 The Wider Landscape
- The wider landscape includes Cisco’s Intersight-managed UCS, Google Distributed Cloud, Azure Local, and DGX Spark’s documented two- and four-unit configurations.Lenovo’s scalable-unit design specifies dedicated head and Kubernetes control-plane nodes before GPU serving.Source: §8.5 · pp. 18–19
8.6 Scope and Caveats
- Cascadia targets client-class machines rather than incumbents’ datacenter design point, and organizations may run both; reported serving measurements cover at most four nodes across separate sessions.The incumbents retain strengths in datacenter serving, multi-tenancy, support, and certifications.Source: §8.6 · p. 19
9 Limitations
The limitations section scopes Cascadia’s trust model, operational dependencies, supported configurations, and the evidence available for performance and functional behavior.
9 Limitations
- Cascadia assumes a single-operator permissioned mesh; tenant namespacing does not establish federation, and CA-hosted admission and fleet management remain centralized dependencies.The state plane subscribes to admitted peers, and its direct-stream design alone does not establish large-fleet scalability.Source: §9 · p. 19
- KV coordination and stream resumption are disabled by default and configuration-dependent; incompatible state falls back to recomputation, while compromised admitted workers remain inside the KV tenant trust boundary.Signed receipts establish provenance rather than numerical correctness, and NAT validation covers simulated configurations.Source: §9 · p. 19
- Performance findings apply to the dated deployments and workloads reported, while chain-transfer and continuation checks establish functional behavior rather than performance or availability guarantees.NPU non-determinism limits bytewise comparison across executors.Source: §9 · p. 19
10 Conclusion
Cascadia combines serving and request handling on client machines, and the reported deployments show higher throughput at multiple fleet sizes while characterizing tested configurations and functional behavior.
10 Conclusion
- Cascadia places ingress, scheduling, and inference on the same client machines, with CA admission and fleet management outside the request path.Whole-model replicas and pipeline chains share an authenticated API, signed response receipts, and local audit logs.Source: §10 · p. 19
- 3.10× response throughput at three nodes relative to one, and 4.06× the throughput of direct single-node serving in a separate four-node deployment, were recorded.Paired latency and runtime observations characterize the tested configurations; functional checks establish the scope of chain and KV behavior.Source: §10 · p. 20
- The reported results support client fleets as a practical serving substrate with a different deployment footprint from managed datacenter clusters.The paper’s repository includes benchmark scripts, curated records, and a claim-to-evidence map.Source: §10 · p. 20