PaperlayerView exact revision on arXiv

Source-linked AI summary

InferSlice: A Reference Architecture for SLO-driven LLM Inference over a Telco Wide-Area Network

Juan Parra-Ullauri, Anthony Chiu, Hari Madhukumar, Zhaozhou Wu, Anderson Bravalheri, Alex Healing, Andy Corston-Petrie, Dimitra Simeonidou, Nektaria Efthymiou

arXiv:2610.32638v1cs.DCcs.AI

TL;DR

Centralised serving and data-centre-oriented frameworks leave limited support for coordinating inference across heterogeneous, geographically distributed telco resources. InferSlice integrates SLO-driven serving, network slicing, GPU partitioning and observability, and its WAN evaluation finds distinct tier regimes and technique-specific operating boundaries.

  • Problem

    Existing LLM serving frameworks are designed for tightly coupled data centres and offer limited support for coordinating geographically distributed, heterogeneous resources.

    Source: §1 · p. 1

  • Method

    InferSlice is a reference architecture that integrates SLO-driven distributed serving, network slicing, GPU partitioning, elastic serving and observability across telco sites.

    Source: §1 · p. 2

  • Results

    Replication is placement-insensitive but memory-bound, pipeline parallelism meets the 500 ms SLO on measured cross-site links, and tensor parallelism and prefill/decode disaggregation are infeasible beyond a single subnet.

    Source: §7 · p. 22

  • Takeaways & Limitations

    The evaluated WAN boundaries indicate that distributed inference techniques should be matched to the network tier they tolerate, while serving models beyond a single site’s memory remains open.

    Source: §7 · p. 22

    Experiments use statically configured policies and evaluate distributed techniques in isolation, so closed-loop control and interactions between network and compute pressure are not established.

    Source: §6.2 · p. 19

Abstract

from arXiv · show

LLM inference is increasingly deployed as a core service for enterprise and interactive applications, but sovereignty, compliance, latency and resource-locality requirements can limit the suitability of centralised cloud deployments. Telco operators offer geographically distributed compute and network infrastructure for addressing these constraints. However, existing LLM serving frameworks are designed for tightly coupled data centres and provide limited support for coordinating geographically distributed and heterogeneous resources. We present InferSlice, a reference architecture for Service-Level Objective (SLO)-driven LLM inference over a telco infrastructure, integrating network slicing, GPU partitioning, elastic serving and observability. Led by BT in collaboration with the University of Bristol and NVIDIA, and realised on the JOINER UK national-scale wide-area network (WAN), InferSlice is evaluated on service differentiation across tiers, elastic scale-out, client-edge bandwidth as a bottleneck, and the reach of tensor, pipeline, data parallelism, and prefill/decode disaggregation over the WAN. Our results show that, on a general-purpose telco network, tiers yield distinct latency-throughput regimes, and weight streaming shortens scale-out time. Replication is placement-insensitive but memory-bound; pipeline parallelism tolerates near- and mid-cross-site hops, while tensor parallelism and disaggregation fail beyond a single subnet hop. We share practical insights from operating SLO-driven inference on a telco network, lessons learned and open questions, and hope to encourage further research on distributed and heterogeneous LLM inference systems.

1 INTRODUCTION

Centralised LLM serving can raise sovereignty, latency and resource-locality concerns, while existing frameworks provide limited support for coordinating heterogeneous telco sites. InferSlice proposes SLO-driven inference across those sites and is evaluated over a shared, general-purpose WAN.

1 INTRODUCTION

  • Telco infrastructure offers geographically distributed compute and networks that can bring inference closer to users and data under operator control.The paper identifies sovereignty, jurisdictional control, shorter network paths and operational predictability as potential benefits.Source: §1 · pp. 1–2
  • Existing serving frameworks target tightly coupled data centres, while telco sites vary in accelerators, latency, bandwidth and resource availability.Different workloads also prioritise different SLOs, so serving configuration, resource allocation and placement must reflect both requirements and available resources.Source: §1 · p. 2
  • InferSlice jointly manages application SLOs, distributed compute and network conditions across telco sites rather than assuming a tightly coupled resource pool or fixed placement.The cloud-native architecture integrates distributed serving, resource orchestration, network management and end-to-end observability.Source: §1 · p. 2
  • InferSlice is deployed on JOINER, a shared UK WAN without dedicated AI interconnects or RDMA, and evaluated across service tiers, scaling and distributed inference.The experiments examine differentiated SLO outcomes under load, scale-out latency, and how WAN latency and bandwidth constrain parallelism and disaggregation.Source: §1 · p. 2

2 BACKGROUND AND RELATED WORK

Distributed LLM inference spans model parallelism, replication and functional disaggregation, each with distinct communication costs and scaling roles. InferSlice addresses a gap in coordinating these strategies with SLOs, compute and network conditions across operator-controlled sites.

2.1 Distributed LLM Inference Fundamentals

  • Distributed inference combines model parallelism, replication and functional disaggregation, which scale a model, independent instances and inference stages, respectively.The dimensions can compose, and this work evaluates TP, PP, DP and prefill/decode disaggregation over a WAN.Source: §2.1 · p. 2, §2.1.3 · p. 3

2.1.1 Model parallelism.

  • Tensor parallelism partitions layer weights but requires frequent collective communication, whereas pipeline parallelism partitions layers and exchanges activations between stages.These strategies commonly require low-latency, high-bandwidth interconnects because synchronisation frequency, rather than site boundaries alone, is the binding constraint.Source: §2.1.1 · pp. 2–3

2.1.2 Replication.

  • Replication deploys independent complete model copies, scaling capacity without inference-time synchronisation between replicas.Replicas can be independently added or removed and requests routed by load, locality or SLO; the independence also provides fault isolation.Source: §2.1.2 · p. 3

2.1.3 Functional disaggregation.

  • Prefill/decode disaggregation separates prompt processing from autoregressive token generation and transfers KV cache between the two services.This allows the functions to be scaled for their different compute characteristics, at the cost of coordination and network dependencies tied to cache size and bandwidth.Source: §2.1.3 · p. 3

2.2 SLO-Aware LLM Serving

  • TTFT measures time to the first token, while TPOT measures latency between subsequent tokens; SLO attainment differs from goodput and raw throughput.For distributed serving, placement also matters because network latency, queueing and capacity vary by site.Source: §2.2 · p. 3

2.3 Telco Infrastructure as an AI Platform

  • Telco infrastructure can extend network service differentiation to AI inference by allocating and placing resources according to application SLOs.Operator-controlled compute sites may provide sovereignty, governance, proximity and predictability; the opportunity is not limited to radio-edge compute.Source: §2.3 · p. 4

2.4 Related Work

  • Prior work addresses industrial serving, telco tiers, WAN disaggregation, network-aware routing and volunteer inference, but not their joint orchestration across operator sites.InferSlice treats serving location and inference configuration as jointly selected according to application needs, site resources and network conditions.Source: §2.4 · pp. 4–5

3 SYSTEM DESIGN

InferSlice combines isolated telco PoPs, tier-specific network and compute allocation, distributed replica management, accelerator-level inference execution, an SLO-oriented control plane and cross-layer telemetry. These layers separate resource enforcement, workload serving and decision-making while allowing them to coordinate across sites.

3.1 Physical Infrastructure Layer

  • The physical layer models PoPs as CPU-, memory-, storage- and GPU-equipped hosts connected over a WAN, using JOINER VMs with L40S accelerators and 10 Gbps NIC links.The deployment uses Dell R660/R760 hosts with up to 64 physical CPU cores and 512 GB RAM.Source: §3.1 · p. 5

3.2 Virtualisation Layer

  • VM and container virtualisation provide isolated, portable workloads across PoPs, with Kubernetes, GPU sharing and persistent storage for data and model weights.The implementation uses VMware, RKE2 and containerd; model weights may be streamed on demand from MinIO.Source: §3.2 · pp. 5, 7

3.3 Network Isolation and Slicing Layer

  • Network slicing maps Free, Standard and Premium tiers to separate logical networks with independently enforced connectivity and bandwidth policies.Geneve preserves inner packet headers for slice QoS classification, while the shared JOINER subnet can become a contention point at high load.Source: §3.3 · p. 7

3.4 Compute Orchestration and Resource Scheduling Layer

  • Compute orchestration assigns each tier GPU and CPU shares, quotas and priorities because network isolation alone cannot guarantee inference service differentiation.Run:ai supports fractional GPU allocation, while KubeRay constrains replica counts to the capacity of each tier’s slice.Source: §3.4 · p. 8

3.5 Distributed Serving and Replica Management Layer

  • Distributed serving expresses workloads as models and replicas, translating application requirements into resource requests while scaling replicas within tier capacity.Ray and KubeRay manage deployment lifecycles, and the Ray autoscaler adjusts replica counts within configured bounds and slice capacity.Source: §3.5 · p. 8

3.6 Inference Optimisation Layer

  • The inference optimisation layer manages model loading, GPU memory, request scheduling, batching and KV-cache execution within each replica.vLLM provides the runtime and supports tensor, pipeline and data parallelism as well as prefill/decode disaggregation.Source: §3.6 · pp. 8–9

3.7 Control Plane

  • The control plane uses SLOs, demand, resource availability, network conditions and observed performance to determine serving configuration and placement.Planning, forecasting, semantic routing and cache reuse inform configuration changes enforced by the underlying layers.Source: §3.7 · p. 9

3.8 Observability Layer

  • The observability layer aggregates compute, network, serving and inference telemetry to monitor SLOs and support runtime decisions and offline analysis.Historical measurements also support forecasting, capacity planning and analysis of operating boundaries across geographically distributed resources.Source: §3.8 · pp. 9–10

4 EXPERIMENTAL SETUP

The evaluation uses JOINER’s geographically distributed compute and network resources to characterize WAN conditions and test SLO-driven serving, scale-out, and distributed inference. Its setup covers service differentiation, client-link constraints, model-parallel placement, bandwidth limits, and prefill/decode disaggregation.

4.1 Testbed and Workload

  • The testbed spans six JOINER PoPs with L40S GPUs, and experiments use ShareGPT requests with models selected for service-tier and distributed-inference tests.The evaluation records TTFT, TPOT, and aggregate token throughput.Source: §4.1 · pp. 10–11

4.2 Network Substrate Characterisation

  • Near- and mid-cross-PoP links are candidates for pipeline parallelism, while distant paths are unsuitable for synchronous strategies and tensor parallelism is confined intra-PoP.Mid-cross-PoP paths remain suitable for pipeline parallelism when inter-stage communication is sufficiently coarse-grained.Source: §4.2 · p. 11

4.3.1 Service Differentiation

  • Free, Standard, and Premium slices share the GPU pool but differ in fractional GPU and CPU allocations, replica ceilings, and tested concurrency.The experiment measures P99 TTFT and goodput against a 500 ms TTFT SLO.Source: §4.3.1 · p. 12

4.3.2 Cold-start latency

  • The cold-start experiment compares default memory-mapped weight loading with parallel streaming during cold deployment and scale-up from minimum replicas.Five replicas are forced under load; readiness and each worker’s first served response are timed.Source: §4.3.2 · pp. 12–13

4.3.3 Network bottleneck identification

  • Network experiments vary client-facing bandwidth and compare two-shard tensor and pipeline parallelism across PoP pairs at approximately 9, 18, and 33 ms RTT.Feasibility is assessed by SLO compliance; four-shard configurations were excluded because they did not initialize within 40 minutes and spanned links with different RTTs.Source: §4.3.3 · p. 13
  • Bandwidth sweeps isolate inter-shard capacity for pipeline and tensor parallelism, while a separate experiment tests prefill/decode disaggregation across PoP pairs.The disaggregation test uses a queue-free 2.7 requests/s ShareGPT workload and evaluates SLO compliance at each placement’s natural bandwidth.Source: §4.3.3 · pp. 13–14

5 EVALUATION

The evaluation finds that tier capacity, weight loading, and WAN conditions shape latency, goodput, scale-out time, and the feasibility of distributed inference.

5.1 Service differentiation across slices

  • At peak, Premium delivers about 2.2× Free’s and 1.7× Standard’s compliant goodput; Standard delivers about 1.3× Free’s.Free breaches the SLO at concurrency 16 and Standard at 20, while Premium stays below it across the sweep except for a transient cold-start excursion.Source: §5.1 · p. 14, Figure 5 · p. 15

5.2.1 Network bottleneck identification

  • PP stays below the 500 ms P99 TTFT SLO on measured paths, but its queue-free rate falls from 4.40 requests/s co-located to 0.85 at 33 ms RTT; TP exceeds the SLO on every tested pair.TP’s maximum queue-free rate is 0.035–0.09 requests/s, with median TTFT of 1.7–4.8 s.Source: §5.2.1 · pp. 15–16, Table 5 · p. 16

5.2 Cold-Start Reduction

  • About 65%: runai-streamer cuts cold deployment from approximately 950 s to 330 s and end-to-end scale-out from approximately 1,500 s to 610 s.Hot-phase medians match cold-phase medians, indicating that storage-cache warmth does not bound weight acquisition.Source: §5.2 · p. 14, Figure 6 · p. 16

5.2.1 Network bottleneck identification

  • At concurrency 10, a 0.5 Mbps cap raises P99 TTFT to 1.9–3.3 s and P99 TPOT to 150–276 ms, while throughput falls to 217–224 tok/s.The transition to bandwidth-limited behavior is bracketed between 0.5 and 1 Mbps at this load.Source: §5.2.1 · pp. 14–15, Figure 7 · p. 17
  • Premium’s unconstrained receive rate rises from 0.91 Mbps at concurrency 10 to 1.32 Mbps at 16 and 2.45 Mbps at 32.Queue saturation predominantly raises TTFT, whereas severe bandwidth saturation reduces link and token throughput and increases both TTFT and TPOT.Source: §5.2.1 · p. 15, Figure 8 · p. 17
  • Disaggregation raises median TTFT from 0.11 s co-located to 4.2–4.4 s across 9–21 ms RTT pairs, while median TPOT is essentially unchanged.KV-cache transfer and NIXL handover dominate the cost, despite transfer rates below path capacity; batching or pipelining is needed to reduce the delay.Source: §5.2.1 · pp. 17–18
  • PP’s median-TTFT-doubling bandwidth threshold falls from 100–200 Mbps at 9 ms RTT to 25–35 Mbps at 33 ms RTT.These thresholds exceed mean one-way traffic demand because bursty pipeline activations must drain before the next burst arrives.Source: §5.2.1 · p. 16, Figure 9 · p. 18

6 DISCUSSION AND OPEN QUESTIONS

InferSlice’s WAN measurements show that each inference strategy has distinct operating boundaries, while SLO attainment depends on identifying the active bottleneck and provisioning ahead of demand. The discussion identifies limits in the evaluation and outlines research directions in heterogeneous placement, composing techniques, stateful workloads, closed-loop control, and energy-aware orchestration.

6.1.1 L1: Communication Patterns Define the Operating Boundaries of Distributed Inference

  • Tensor parallelism is confined to a single subnet; pipeline parallelism meets the latency SLO at measured cross-PoP pairs but is feasible only at the Mid tier under the stated capacity criterion, while prefill–decode disaggregation is infeasible at all cross-PoP tiers.Replication is placement-insensitive but requires the model to fit in one node’s GPU memory; pipeline parallelism sustains 59%, 35% and 19% of co-located capacity at 9, 18 and 33 ms.Source: §6.1.1 · p. 18, Table 7 · p. 20, §6.2 · p. 20

6.1.2 L2: Bottlenecks Shift with Load

  • The binding resource shifts with load: queue saturation raises TTFT while TPOT stays near its unconstrained value, whereas an edge-bandwidth cap clamps throughput and raises both metrics.The Premium tier needs 0.91, 1.32 and 2.45 Mbps at concurrency 10, 16 and 32, respectively; adding replicas can relieve queueing but not a bandwidth bottleneck.Source: §6.1.2 · pp. 18–19

6.1.3 L3: Cold Start Limits Elastic SLO Enforcement

  • Parallel weight streaming reduces end-to-end scale-out from approximately 1,500 s to 610 s, but new replicas still take minutes to serve a first response, too late for measured SLO breaches.Readiness takes about two minutes, and the 8B-model cold-start lead time is expected to grow with model size; the discussion recommends provisioning ahead through forecasting, warm standbys or faster weight distribution.Source: §6.1.3 · p. 19

6.2 Scope and Limitations

  • The evaluation uses short-lived stateless ShareGPT chat requests on homogeneous L40S GPUs and does not test closed-loop control, simultaneous network and compute pressure, or model-parallel configurations beyond two shards.The discussion notes that longer-context, stateful agentic workloads may alter operating boundaries, particularly for disaggregation, as KV-cache transfer costs grow with context.Source: §6.2 · p. 19

6.3.1 Heterogeneity in compute and network

  • Telco sites vary in accelerator generation, memory and headroom, while WAN paths vary substantially in latency and throughput; the authors call for placement and routing policies that jointly consider compute capability and network conditions.The feasibility of individual techniques does not establish that their compositions are feasible; interactions such as pipeline bubbles under replica autoscaling remain open questions.Source: §6.3.1 · p. 20, §6.3.2 · p. 20

6.3.3 Serving models that exceed single-node memory over the WAN

  • Over the WAN, communication on the critical path is the main barrier to scaling beyond single-node memory: per-layer all-reduce confines tensor parallelism to a subnet, while pipeline parallelism loses sustainable load as RTT rises.Moving 15–35 MB of KV-cache per request adds seconds to TTFT despite using a small fraction of available bandwidth; the authors identify communication overlap, activation compression, and pipelined KV-cache transfer as research directions.Source: §6.3.3 · p. 20

6.3.4 Stateful and agentic workloads

  • For stateful agentic workloads, routing may need to favor a site holding the relevant KV-cache over one with more spare capacity; the evaluation does not cover these workloads.The discussion calls for task-level SLOs and policies that balance cache affinity with load and failure recovery.Source: §6.2 · p. 19, §6.3.4 · p. 21

6.3.5 SLO-aware placement and control

  • Because the evaluation uses static per-slice policies and the binding resource changes with load, the open control problem is closing the loop between observation, prediction and resource control.Energy, carbon and site power availability add objectives and constraints to placement and orchestration decisions.Source: §6.3.5 · p. 21, §6.3.6 · p. 21

7 CONCLUSION

InferSlice combines slicing, GPU partitioning, elastic serving, and observability to run SLO-driven inference across JOINER sites. Its results show distinct service regimes and communication-dependent limits on WAN-distributed inference.

7 CONCLUSION

  • 2.2× peak SLO-compliant goodput separated Premium and Free tiers, while parallel weight streaming reduced scale-out time from approximately 1,500 s to 610 s.The deployment spanned six PoPs on a shared network without RDMA or specialised inter-site interconnects.Source: §7 · p. 21
  • Queue saturation and bandwidth saturation leave distinct TTFT and TPOT signatures, making cross-layer observability a prerequisite for SLO enforcement.The binding resource shifts with load.Source: §7 · p. 22
  • Replication is placement-insensitive but memory-bound; pipeline parallelism meets the 500 ms SLO on measured cross-site links, while tensor parallelism and disaggregation fail beyond one subnet.Pipeline parallelism sustains less load as RTT grows, and disaggregation is limited by KV-cache handover rather than propagation delay or bandwidth.Source: §7 · p. 22
  • The evaluation used statically configured policies, stateless chat workloads, and a single accelerator generation.Closed-loop control, stateful and agentic workloads, heterogeneous accelerators, and energy-aware placement remain future directions.Source: §7 · p. 22

A JOINER: TELCO WAN TESTBED

JOINER is a UK research testbed whose institutional network expanded to an international node in Ireland. Its connectivity includes JANET’s national backbone and international research-network links.

A JOINER: TELCO WAN TESTBED

  • JOINER began as a research project in December 2023, started public operations in 2025, and initially brought together 13 UK universities and research laboratories.The first international node, in Ireland, was announced in 2025.Source: A JOINER: TELCO WAN TESTBED · pp. 23–24
  • Figure 10 depicts JANET’s backbone and international research-network links, with nominal link capacities of 100 to 800 Gbit/s.Source: Figure 10 · p. 24
  • JOINER’s listed institutions include universities in Ireland, Northern Ireland, Scotland, and England.The supplied passage lists Trinity College Dublin, Queen’s University Belfast, Edinburgh, Cambridge, UCL, Imperial College London, and Southampton.Source: A JOINER: TELCO WAN TESTBED · p. 24

A.1 Architecture

The appendix describes JOINER’s site resources and network connectivity, then outlines a dashboard for interacting with and monitoring InferSlice across service tiers.

A.1 Architecture

  • JOINER sites include Dell servers with up to 64 physical cores, 512 GB+ RAM, and Nvidia L40S GPUs, while a central Brain aggregates control and management.Source: §A.1 · p. 24

A.2 Connectivity

  • JOINER combines JANET and NDFF connectivity with a 10–40 Gbit/s logical mesh delivered as SD-WAN over L2-VPN, supporting isolated VLAN overlays.Source: §A.2 · pp. 25–26

B INFERSLICE DASHBOARD DESCRIPTION

  • The dashboard lets users generate traffic and compare tier-level latency, throughput, bandwidth, resource use, and UK network topology in real time.Its GPU-allocation view supports fractional allocation, with the Premium tier able to occupy the largest GPU share.Source: B INFERSLICE DASHBOARD DESCRIPTION · p. 26

C.1 Concurrency-policy differentiation

Concurrency policy separates inference into latency-, balance-, and throughput-oriented operating regimes. Higher throughput comes with a latency trade-off, especially for C3 under increasing load.

C.1 Concurrency-policy differentiation

  • C3 sustained 2,117–2,130 tokens/s, approximately 2.4× C2 and 3.0× C1, while C2 reached 883–902 tokens/s, 26–29% above C1.C1 sustained approximately 698–699 tokens/s.Source: §C.1 · p. 26
  • Throughput remained stable after saturation and no requests failed, indicating full GPU utilisation with additional load absorbed through queuing.Source: §C.1 · p. 26
  • C3’s P99 TTFT rose from 308 ms at 8 RPS to 1,305 ms at 12 RPS and 2,001 ms at 32 RPS, about four times the interactive-service threshold.By contrast, C2’s P99 TTFT remained below the 500 ms threshold at all tested request rates.Source: §C.1 · p. 26
  • C1 kept mean TTFT between 99 and 101 ms and P99 TTFT between 130 and 138 ms across loads, while C2 traded higher latency for throughput.TPOT varied by less than 4% within each configuration; the observed latency degradation primarily reflected request admission and queuing effects.Source: §C.1 · p. 26
  • C1 is latency-first, C2 balanced, and C3 throughput-first, allowing distinct service regimes over a common compute and network resource pool.Source: §C.1 · p. 29

C.2.1 Experiment N1 results: End client bandwidth limiting

At concurrency 10, client-edge bandwidth below approximately 1 Mbps becomes a bottleneck; at higher concurrency, the boundary shifts with demand and queueing or compute-side saturation can also limit tiers.

C.2.1 Experiment N1 results: End client bandwidth limiting

  • At concurrency 10, a 0.5 Mbps cap cuts throughput to 216–224 tok/s and raises P99 TTFT to 1.9–3.3 s, below the approximately 1 Mbps bandwidth floor.Above the floor, measured receive rates remain around 0.9–0.94 Mbps, throughput around 370–380 tok/s, and P99 TTFT below the 500 ms SLO.Source: §C.2.1 · p. 29, Table 9 · p. 30
  • At higher concurrency, Standard and Free already violate the TTFT SLO under effectively unconstrained bandwidth, indicating queueing or compute-side saturation before network saturation.Severe bandwidth restriction compounds degradation; unlike queue saturation, it reduces receive and token throughput while increasing both TTFT and TPOT.Source: §C.2.1 · p. 29, Table 10 · p. 31, Table 11 · p. 31
  • As concurrency rises, Premium’s unconstrained demand grows from approximately 0.91 Mbps at 10 to 1.32 Mbps at 16 and 2.45 Mbps at 32, while 0.5 Mbps caps receive rates at 0.45–0.47 Mbps.At concurrency 32, that cap collapses throughput to roughly 221–231 tok/s, with Premium P99 TTFT at 12.8 s and P99 TPOT at 1.19 s.Source: §C.2.1 · p. 29, Table 11 · p. 31, Table 10 · p. 31
Loading 2610.32638v1…