PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Inference Prefill with High-Radix Photonic Interconnects

Arulselvan Madhavan, Peter Carson, Taylor Groves, Thomas Graham

arXiv:2609.01821v1cs.DCcs.AR

TL;DR

The paper addresses how inference prefill can support both high-concurrency throughput and increasingly long contexts despite electrical scale-up limits. It models production-grade MoE workloads across GPU configurations and compares electrical baselines with projected optical interconnects. The results show the largest benefits when communication limits or scale-up-pod boundaries dominate, while the analysis remains prefill-centric and analytical.

  • Problem

    Inference prefill must balance high-concurrency throughput with 128K–1M+ token contexts, but electrical interconnect limits constrain bandwidth, reach, and scale-up pod size.

  • Method

    The paper uses an XLA-based MLIR cost model to evaluate three MoE context scales across four GPU architectures and compare electrical baselines with projected optical configurations.

  • Results

    Optical interconnects improve prefill latency across contexts and platforms, reaching 2.2–4.5× on production platforms at 1M tokens and up to 8.5× for speculative R4.

  • Takeaways & Limitations

    Optical scale-up matters most when workloads cross into lower-bandwidth scale-out fabrics or when faster arithmetic makes communication the limiting component.

  • Takeaways & Limitations

    The results are analytical projections assuming ideal 4× scale-up bandwidth and omit additional link latency, thermal limits, signal integrity, deployment cost, and full TCO; the study is prefill-centric.

Abstract

from arXiv · show

With the rise of inference as today's dominant AI workload, the industry is transitioning to high-bandwidth photonic interconnects to meet the large scale-up requirements of increasingly complex Mixture-of-Experts (MoE) models. This paper quantifies the benefits of 3D-integrated photonic interconnects for inference prefill by analyzing tradeoffs between high-concurrency throughput for Large Language Model (LLM) chat and the large context windows typically required for reasoning and agentic AI. We simulate three MoE models: short context (1K--8K tokens), medium context (128K tokens), and long context (1M tokens). We evaluate this workload across existing copper-based GPU systems and one with high bandwidth integrated photonics. We show 2.1--3.2x latency improvements in the stressed high-batch regimes and 2.8--5.8x improvements over baselines in communication-limited configurations. 3D photonics enable the 1152-GPU footprint required to lower time-to-first-token, yielding 2.2--4.5x speedups across production-grade platforms when electrical systems cross their inherent scale-up-pod limits.

I. Introduction

Inference has become a dominant workload whose prefill phase must balance high-concurrency chat throughput with long contexts for reasoning and agentic applications. The paper examines how photonic bandwidth and larger scale-up pods address these constraints.

  • Production agentic coding workflows have a median initial prefill context of approximately 96K tokens, with nearly half of requests exceeding 128K.
  • Electrical interconnect reach and bandwidth limitations make communication a primary bottleneck for large-scale distributed inference, especially as batches increase.Passive copper reach shrinks to approximately 1m at 224G and tens of centimeters at 448G, restricting high-performance pods to one rack.
  • Photonic interconnects expand high-bandwidth scale-up pods to hundreds or thousands of devices, supporting larger batches and long-context reasoning across multiple racks.The paper highlights radix up to 1,152 and the ability to process 128K+ token contexts without scale-out penalties.
  • 2.2–2.9× prefill-latency improvements are reported across 1K–8K, 128K, and 1M-token contexts on NVIDIA B300 systems spanning 72 to 1,152 processors.The reported ratios depend on batch size, context length, and device count relative to the native scale-up pod; compute-bound configurations show smaller gains.

A. LLM Inference Workload Characteristics

LLM inference alternates between compute-intensive prefill and memory-bandwidth-bound decode, while serving systems must trade aggregate throughput against interactive latency and manage long-context memory pressure.

  • Prefill and Decode: Prefill processes the full prompt in parallel to generate the KV cache and primarily stresses dense matrix computation and GPU FLOPS.Its key service metric is TTFT.
  • Prefill and Decode: Decode generates output tokens autoregressively, loading model weights and the active KV cache for each token.Decode is fundamentally constrained by HBM bandwidth, with TPOT measuring generation fluidity.
  • High-performance serving must balance throughput, interactivity SLAs, and memory demands across these divergent inference phases.
  • Batching Trade-off: Larger batches improve aggregate throughput and provider cost efficiency but increase queuing and iteration times, degrading individual inter-token latency.
  • Memory Pressure: Long-context KV caches consume substantial memory, limiting concurrent users and potentially forcing eviction or slower CPU-memory offload when HBM capacity is exhausted.

2) Cache Pressure and Eviction:

Inference scalability is constrained by electrical interconnect bandwidth, reach, and chip shoreline limits, while photonic integration expands the scale-up domain and mitigates these bottlenecks.

  • Algorithmic methods such as continuous batching and disaggregated inference do not remove the physical limitations of the underlying interconnects.
  • Chip shoreline constraints limit the perimeter available for scale-up SerDes because substantial area is reserved for HBM.This restricts aggregate bandwidth for the large batches needed for efficiency.
  • Electrical scale-up pods are effectively limited to approximately 72–144 GPUs because passive reach is about 1 meter at 224 Gb/s and longer reach requires power-hungry retimers.
  • Scale-out fabrics connect more devices but provide much lower per-GPU bandwidth than scale-up fabrics, so they do not replace a larger scale-up domain.
  • 3D-integrated photonics decouple I/O from package shoreline constraints, increasing bandwidth density and radix while enabling scale-up pods beyond 1,152 GPUs.

A. 3D Photonic Interconnects

3D photonic interconnects stack electronic and photonic components to distribute I/O across the chip, increasing bandwidth and radix while reducing data-movement energy for modeled inference systems.

  • Architecture: 3D-integrated photonics stack host ASICs or SerDes directly onto optical engines, distributing I/O across the chip rather than along the package shoreline.
  • Extreme Bandwidth: A Passage-enabled GPU can support greater than 64 Tb/s bidirectional bandwidth, compared with 14.4 Tb/s for current NVIDIA Blackwell configurations.
  • High Radix: Optical fibers and bidirectional signaling provide an 8X radix increase over copper-based approaches and allow scale-up pods to extend beyond a single rack.
  • Energy Efficiency: The complete 3D-photonic link budget is approximately 4.3 pJ/bit, compared with approximately 5 pJ/bit for passive copper and more than 20 pJ/bit for traditional pluggable optics.
  • Evaluation Approach: The paper quantifies photonic-interconnect benefits using MLIR-based modeling that captures tensor, expert, and context parallelism together with compute and communication costs.

B. Model Configurations

The paper evaluates three MoE model scales and four GPU architectures, using FP4 and FP8 variants while modeling electrical scale-up and scale-out constraints. Matching optical configurations preserve compute and HBM, provide 4× scale-up bandwidth, and extend optical scale-up to 1152 GPUs.

  • Model Configurations: Three evaluated MoE tiers are Mini with 21B active parameters, R1 with 42B, and Next with 201B.All models use Multi-Head Latent Attention, which reduces KV-cache size through low-rank compression relative to GQA-based architectures.
  • Model Configurations: The study evaluates both FP4 and FP8 quantization precisions.The R1 model is a divisibility-adjusted DeepSeek-R1 variant used to satisfy mesh-divisibility constraints across device counts.
  • Hardware Configuration: Four GPU architectures are modeled using per-GPU compute, scale-up pod size, scale-up bandwidth, and cross-rack scale-out bandwidth.Rubin is modeled with up to 72 GPUs per scale-up pod, while speculative R4 is extrapolated to 576 GPUs; R4 specifications are projections.
  • Optical Comparison: Each electrical baseline is paired with an optical configuration having identical per-GPU compute and HBM, 4× scale-up bandwidth, and a maximum 1152-GPU optical pod.The comparison isolates interconnect bandwidth and radix rather than vendor-specific optical products.
  • Evaluation Method: The experiments sweep context lengths of 1K, 8K, 128K, and 1M tokens while varying valid device meshes or batch sizes.Device-sweep batch sizes keep total input tokens near 8M across context lengths.

E. Performance comparisons and table entries

The evaluation measures overlapped prefill latency across context lengths and operating regimes, emphasizing communication-heavy short-context batching and scale-out at 128K tokens. Optical interconnects reduce communication time and preserve scaling, with especially large gains when electrical systems cross rack or scale-up-pod limits.

  • Performance Method: Overlapped prefill latency includes compute–communication overlap, and ratios above 1 indicate lower latency with optical interconnects.The reported ratio is electrical-baseline latency divided by optically interconnected latency for each evaluated configuration.
  • Workload Regimes: The study covers short-context 1K–8K, medium-context 128K, and long-context 1M workloads, which stress communication saturation, computation, and multi-rack scale-out differently.The main results use the R1 42B-active-parameter model in FP4 on NVIDIA B300 unless otherwise stated.
  • Short-Context Workloads: At short contexts, large batches increase GPU utilization but also enlarge activation exchanges, allowing communication to dominate computation.For 8K tokens at batch 2048 on 72 devices, the electrical baseline spends approximately 40.6 seconds communicating versus 16.8 seconds computing.
  • Short-Context Workloads: At the same 8K-token, batch-2048 point, optical interconnects reduce communication time to approximately 10.2 seconds while compute remains 16.8 seconds.The optical configuration uses 4× higher bandwidth than the electrical comparison.
  • Medium-Context Workloads: At 128K tokens on 72 GPUs, computation takes approximately 56.1 seconds versus 20.3 seconds for communication, motivating scale-out to reduce TTFT.The evaluation uses a 288-device batch sweep for the 128K sequence length.
  • Medium-Context Workloads: At 288 GPUs, electrical communication rises to approximately 35.5 seconds while compute falls to 14.3 seconds, whereas optical communication drops to 2.3 seconds.At 288 devices, this produces approximately 2.8–3.0× improvement for B300; rack-limited B200 and Rubin reach approximately 4.3–5.8×, while R4 reaches approximately 2.0–2.1×.

C. Large-Context Workloads (1M Tokens)

For 1M-token prefill, electrical systems encounter rising communication costs as device counts cross rack-scale limits, while optical scale-up sustains scaling across up to 1152 devices. This reduces communication latency and lowers prefill latency for large-context workloads.

  • Component Balance: The optical configuration lowers communication time while leaving compute time unchanged relative to the electrical baseline.The reduced communication component moves communication-limited cases toward compute-limited operation.
  • 1M-Token Scaling: 1152 devices reduce compute time to 26.2 seconds per pass for the 1M-token workload.The analysis sweeps up to 1152 devices at batch 8.
  • Communication Scaling: 39.2 seconds of electrical communication latency at 1152 devices falls to 1.6 seconds with optical scale-up.The optical configuration keeps all 1152 devices in one high-bandwidth scale-up pod.
  • Platform Results: 2.27× is the best device-sweep improvement for B300 at 1152 devices, while B200/Rubin reach approximately 3.4–4.5× and R4 exceeds 8×.The larger gains reflect latency floors set more by scale-up limits than per-GPU FLOPS alone.
  • Batch Scaling: Optical interconnects provide greater per-GPU throughput and lower prefill latency as batch size increases.This regime is relevant to agentic and reasoning workloads that generate many intermediate tokens per request.

D. Summary Across Models, Precisions, and Hardware

Across models, precisions, and hardware, optical gains depend primarily on whether collective communication lies on the overlapped-latency critical path. Benefits are modest in compute-bound cases and peak when workloads exceed the native electrical scale-up pod.

  • Sweep Method: Table IV reports the best electrical-to-optical overlapped prefill-latency ratio across 36 representative FP4 scenarios.The sweep spans three model scales, four hardware platforms, sequence lengths, device counts, and batch sizes.
  • Hardware and Precision: At 1K–8K contexts, optical benefits generally increase from B200 to B300 to Rubin to R4 as faster GPUs expose communication costs.FP4 shows larger multipliers than FP8 for the same reason.
  • Context and Token Tiers: 8K gains can be lower than 1K gains because the derived batch size falls eightfold at the same total-token tier, making communication less dominant.Optics do not provide uniform speedup when the baseline is not communication-limited.
  • Scale-Up-Pod Transitions: At 128K, rack-limited platforms show substantially larger gains at 288 devices than at 16 or 72 devices after crossing into slower scale-out communication.The same scale-up-boundary mechanism appears at larger scale for 1M-token workloads.
  • Hardware Generation: B300 shows slightly smaller optical multipliers than compute scaling alone predicts because its higher modeled scale-out bandwidth reduces the electrical penalty at 288 devices.The relevant variable is the balance among compute, scale-up bandwidth, and scale-out bandwidth.
  • Rubin and R4: R4 reaches approximately 8.0–8.5× at 1M tokens and 1152 devices because its 576-GPU native pod is insufficient for that footprint.At 128K and 288 devices, its larger native pod instead yields smaller gains of approximately 2.0–2.1×.
  • Across Model Scales: Communication on the overlapped-latency critical path is the dominant predictor of optical-interconnect gains across Mini, R1, and Next.The qualitative structure remains stable across model scales, although the exact multiplier varies with model size.

E. Serving-Level DES Validation

The serving-level DES validation shows optical scale-up reducing p99 TTFT, while decode saturation can offset these gains in end-to-end latency. The results therefore position photonics primarily as a prefill lever whose system impact depends on decode capacity.

  • Serving-Level DES Validation: 12–20% lower p99 TTFT results from shorter prefill steps and faster queue drainage across tested concurrencies.At concurrency 6144, p99 TTFT falls from 81.8 s to 65.5 s.
  • Serving-Level DES Validation: 77–110% higher p99 TPOT occurs when faster prefill pushes the single decode worker toward its batch cap.At concurrency 6144, p99 TPOT rises from 31.6 ms to 66.3 ms as decode batches grow from approximately 694 to 734.
  • Serving-Level DES Validation: Approximately neutral p99 end-to-end latency at OSL 1024 reflects cancellation between TTFT improvement and TPOT inflation.At concurrency 6144, end-to-end latency changes from 103.4 s to 102.0 s.
  • Serving-Level DES Validation: Optical scale-up provides TTFT and input-throughput headroom, but converting it into end-to-end gains requires co-designing decode capacity.Figure 7 evaluates TTFT, TPOT, and end-to-end latency for electrical and optical systems at annotated in-flight concurrencies.

VI. Related Work

The paper situates photonic interconnects against electrical scale-up and scale-out fabrics whose power, reach, and signal-integrity limits constrain large distributed inference. Its modeled results indicate that optical benefits are largest when workloads would otherwise cross scale-up-pod boundaries.

  • VI. Related Work: Electrical links face worsening power, reach, and signal-integrity limits, while CXL and UCIe do not provide the rack-spanning GPU-to-GPU bandwidth studied here.
  • VI. Related Work: The evidence is modeled across three MoE model scales and four hardware setups rather than measured on deployed photonic hardware.
  • VI. Related Work: 2.1–2.9× prefill improvements appear at 1K–8K tokens in high-batch configurations where FP4 arithmetic shifts the critical path toward communication.
  • VI. Related Work: Approximately 4.3–5.8× gains at 128K tokens occur on rack-limited platforms when 288+ GPUs remain inside a high-bandwidth scale-up pod.
  • VI. Related Work: 2.2–4.5× modeled improvements at 1M tokens on production platforms rise to up to 8.5× for the speculative R4 configuration.

Appendix A DeepSeek R1 Model Configuration

The appendix defines a modified DeepSeek-R1 configuration and a three-axis parallelism strategy for evaluating prefill meshes. Meshes are selected by minimum overlapped prefill latency for each device count and context length.

  • DeepSeek R1 Model Configuration: The modified DeepSeek-R1 uses embedding dimension 8064, 144 attention heads, 288 routed experts, and vocabulary size 145,440.These choices enforce divisibility across the device meshes used in the sweep.
  • DeepSeek R1 Model Configuration: The model configuration is approximately 14% larger than production DeepSeek-R1 while retaining representative communication patterns and scaling behavior.
  • Parallelism Strategy: The physical mesh uses x for sequence/context parallelism, y for expert and tensor parallelism, and z for FFW tensor parallelism.
  • Parallelism Strategy: The y-axis is the most bandwidth-intensive because it carries expert routing all-to-all traffic and attention all-gather/reduce-scatter operations.
  • Parallelism Strategy: For 288 devices, the preferred mesh is 2×144×1, allocating the largest dimension to y.Table V selects each mesh from hundreds of candidates using minimum overlapped prefill latency.

Appendix C GB200 Specifications

The appendix specifies modeled GPU compute and memory assumptions, quantization multipliers, and percentile measurements for a B300 FP4 disaggregated serving scenario. The reported latency distributions compare electrical and optical configurations across TTFT, TPOT, and end-to-end latency.

  • GB200 Specifications: Each modeled GPU has 192 GB HBM3e and 8 TB/s memory bandwidth, with compute derived from the OpenXLA tcgen05.mma instruction.
  • GB200 Specifications: FP8 and FP4 compute rates are modeled as 2× and 4× the BF16 rate, respectively.
  • GB200 Specifications: Table VI presents FP8 first and FP4 second within each model–hardware block, using the same column definitions as Table IV.
  • Serving Measurements: The B300 FP4 DES evaluation uses six 8-GPU prefill workers and one 24-GPU decode worker under Poisson arrivals at λ=250 req/s.The scenario uses ISL 8192 and OSL 1024.
  • Serving Measurements: Latency distributions report mean, median, and p99 for TTFT, TPOT, and end-to-end latency, with electrical and optical configurations distinguished by color.
Loading 2609.01821v1…