PaperlayerView exact revision on arXiv

Source-linked AI summary

Expanding across time to deliver bandwidth efficiency and low latency

William M. Mellette, Rajdeep Das, Yibo Guo, Rob McGuinness, Alex C. Snoeren, George Porter

arXiv:1903.12307v1cs.NI

TL;DR

Datacenter networks need both low latency and high bandwidth, but static topologies limit capacity or impose bandwidth tax, while reconfigurable topologies delay latency-sensitive traffic. Opera combines time-varying expander graphs for immediate multi-hop forwarding with direct circuits over time for bulk traffic. It delivers up to 4× higher shuffle throughput and supports 60% higher load than cost-equivalent static networks without adversely impacting short-flow completion times.

  • Problem

    Datacenter networks must provide low-latency and high-bandwidth delivery despite the capacity limits, bandwidth tax, and reconfiguration delays of existing designs.

  • Method

    Opera rapidly and deterministically reconfigures circuits so each instantaneous topology is an expander while every endpoint pair receives periodic direct connectivity over time.

  • Results

    Opera delivers up to 4× higher shuffle throughput and 60% higher supported load than cost-equivalent static networks without adversely impacting short-flow completion times.

  • Takeaways & Limitations

    A single Opera fabric supports both bulk and low-latency traffic instead of requiring separate networks for the two traffic classes.

  • Takeaways & Limitations

    Existing reconfigurable approaches can impose 10–100s of milliseconds of delay while circuits are provisioned, motivating Opera’s separate immediate expander paths.

Abstract

from arXiv · show

Datacenters need networks that support both low-latency and high-bandwidth packet delivery to meet the stringent requirements of modern applications. We present Opera, a dynamic network that delivers latency-sensitive traffic quickly by relying on multi-hop forwarding in the same way as expander-graph-based approaches, but provides near-optimal bandwidth for bulk flows through direct forwarding over time-varying source-to-destination circuits. The key to Opera's design is the rapid and deterministic reconfiguration of the network, piece-by-piece, such that at any moment in time the network implements an expander graph, yet, integrated across time, the network provides bandwidth-efficient single-hop paths between all racks. We show that Opera supports low-latency traffic with flow completion times comparable to cost-equivalent static topologies, while delivering up to 4x the bandwidth for all-to-all traffic and supporting 60% higher load for published datacenter workloads.

1 Introduction

Datacenter networks must balance rising bandwidth demands with low-latency delivery, but existing under-provisioned and dynamic designs trade one objective against the other. Opera addresses this tension by combining time-varying expander connectivity for latency-sensitive traffic with direct circuits for bulk traffic.

  • Rising host link rates make preserving full-bisection “big-switch” bandwidth increasingly costly and potentially infeasible.
  • Under-provisioned topologies reduce cost but bias performance toward particular workloads, while dynamic reconfiguration can delay latency-sensitive traffic.
  • Existing dynamic networks face a trade-off between amortizing reconfiguration overhead and avoiding inefficient network configurations.
  • Opera continuously reconfigures ToR uplinks through time-varying expander graphs without runtime circuit selection or network-wide demand collection.
  • Opera forwards latency-sensitive traffic indirectly over a low-diameter network while periodically providing direct links for bandwidth-efficient bulk traffic.
  • 4× higher shuffle throughput and up to 60% higher throughput on published skewed workloads are reported relative to cost-equivalent static topologies.For the published skewed workloads, Opera maintains equivalent flow completion times across all flow sizes.

2 Network efficiency

Datacenter workloads are highly heterogeneous: most bytes belong to bulk flows, while short flows are constrained by propagation delay. Network designs therefore trade capacity, bandwidth tax, and reconfiguration delay through different combinations of oversubscription, indirection, and dynamic links.

  • 2.1 Workload properties: The vast majority of transmitted bytes are in bulk flows, motivating low bandwidth tax without substantially increasing short-flow propagation delay.
  • 2.1 Workload properties: Small-flow FCT is constrained by propagation delay, whereas bulk-flow FCT is governed by available path capacity.
  • 2.3 Reduced capacity networks: Oversubscribed folded-Clos networks provide direct routing but only 1/M = u/d of fully provisioned bandwidth.
  • 2.3 Reduced capacity networks: Expander networks incur 300–400% bandwidth tax for typical 4–5-hop paths, partly offset by provisioning more upward than downward ports.
  • 2.3 Reduced capacity networks: Reconfigurable topologies can create single-hop paths and avoid bandwidth tax when matched to demand, but provisioning links introduces delay and unavailable circuits.
  • 2.3 Reduced capacity networks: Existing reconfigurable proposals impose end-to-end delays on the order of 10–100s of milliseconds and therefore commonly require a separate packet-switched network for latency-sensitive traffic.

3 Design

Opera combines time-varying expander connectivity with reconfigurable direct paths, allowing low-latency packets to use immediate multi-hop routes while bulk traffic waits for bandwidth-efficient one-hop connections.

  • Design principles: Opera combines expansion for short paths with reconfigurability to reduce bulk traffic’s bandwidth tax.Expander graphs provide short paths, while circuit switches establish direct rack-to-rack paths over time.
  • Continuous connectivity: Staggered reconfigurations preserve continuous connectivity by ensuring that not all circuit switches change simultaneously.Offsetting switch changes avoids the global disruption caused by simultaneous reconfiguration.
  • Time-varying topology: At every instant, the active circuit combination forms an expander graph, while over a cycle every rack pair receives a direct path.This supports multi-hop forwarding for latency-sensitive traffic and direct forwarding for bulk traffic.
  • Forwarding: In the example topology, racks 1 and 8 can use either an immediate two-hop path or a delayed one-hop path when their direct matching is instantiated.The two paths illustrate the trade-off between latency and bandwidth efficiency.
  • Forwarding: Opera lets each packet choose between immediate indirect forwarding over the current expander and delayed direct forwarding to avoid the bandwidth tax.The forwarding choice can be made without runtime circuit-assignment selection or network-wide demand collection.

4 Implementation

Opera’s implementation sets timing and transport mechanisms so packets can traverse changing topologies safely, while classifying traffic according to whether it can wait for direct connectivity.

  • Traffic classes: Opera defines low-latency traffic as traffic that cannot wait for a direct bandwidth-efficient path to become available.The boundary depends on how quickly circuit switches cycle through direct matchings.
  • Timing: 10 µs-scale circuit reconfiguration delays and end-to-end delay constrain how quickly Opera can cycle through configurations.Circuit amortization and packet traversal safety jointly determine the cycle time.
  • Timing: Opera spaces reconfigurations by ε+r, where ε is worst-case end-to-end delay and r is circuit-switch reconfiguration delay.Packets sent during a topology slice avoid the circuit that is about to reconfigure, giving them at least ε time to traverse the network.
  • Timing: With 24 KB queues, five-hop worst-case paths, 500-ns per-hop propagation, and 10-Gb/s links, Opera sets ε to 90 µs and achieves a 10.7 ms cycle time.These parameters yield a 98% duty cycle and place flows ≥15 MB within a factor of 2 of ideal completion time.
  • Transport: Priority queuing lets NICs and ToRs send low-latency traffic immediately while delaying bulk traffic until direct connectivity is available.The design avoids head-of-line blocking between the two service modes.
  • Transport: Opera uses shallow queues and identifies NDP as a promising low-latency transport, while leaving Homa integration for future work.The implementation reports that 12-KB queues work well for Opera.

5 Evaluation

Opera is evaluated against static and reconfigurable networks across mixed, bulk, low-latency, failure, scale, and cost scenarios. It improves bandwidth for bulk-heavy workloads while retaining competitive low-latency performance, but its indirect-path capacity is limited for latency-only traffic.

  • 5.1 Real-world traffic: Opera services 40% load, while static networks begin saturating past 25% load.Opera pays bandwidth tax on only 4% low-latency traffic, yielding an 8.4% effective aggregate bandwidth tax.
  • 5.2 Bulk traffic: 60 ms is Opera’s 99th-percentile FCT for 100-KB all-to-all Shuffle, versus 227 ms for 3:1 Clos and 223 ms for an expander.Opera carries the shuffle flows over direct, bandwidth-tax-free paths.
  • 5.3 Only low-latency flows: All networks provide equivalent Websearch FCTs through 10% load, but Opera cannot admit additional load beyond 10%.The folded Clos and expander saturate slightly above 25% load, while their FCTs become nearly 100× worse than at 1% load.
  • 5.4 Mixed traffic: Opera delivers up to 4× more throughput than static topologies at low Websearch load and almost 2× more at 10% load.The trade-off is a factor-of-2 reduction in low-latency capacity for a factor-of-2–4 increase in bulk capacity.
  • 5.5 Fault tolerance: Opera withstands about 4% link failures, 7% ToR failures, or 33% circuit-switch failures without connectivity loss.Its fault tolerance stems from the properties of expander graphs.
  • 5.6 Network scale and cost sensitivity: For permutation and moderately skewed traffic, Opera outperforms folded Clos and expanders when α < 1.8, while shuffle throughput is 2× higher even at α = 2.With a single hot rack, Opera offers performance comparable to a static expander.

6 Prototype

The prototype examines Opera’s low-latency behavior when packets share switches with bulk traffic and measures the implementation’s routing-state requirements. It uses virtualized ToR and circuit switches in a single Tofino device with periodic controller-driven reconfiguration.

  • 6 Prototype: The prototype uses eight ToRs with four uplinks each, connected to four emulated circuit switches inside one 6.5-Tb/s Tofino switch.A P4 program emulates circuit-switch forwarding, while a control server updates switch state and signals reconfiguration to hosts.
  • 6.1 End-to-end latency: Without bulk traffic, prototype ping latency reaches up to 9 µs, with about 3 µs of P4 forwarding delay per hop.The measured latency also includes path length and RoCE/MPI variability.
  • 6.1 End-to-end latency: Low-latency packets can temporarily buffer behind bulk packets already exiting a switch, making the remaining wait effectively random.This effect produces a smooth latency distribution in the prototype.
  • 6 Prototype: Opera’s routing-state requirement grows as O(N_rack^2) entries because each topology slice has N_rack destinations.The ruleset-size study measures resulting switch-memory utilization across datacenter sizes.

7 Related work

Related work spans static sparse topologies, reactive and deterministic dynamic networks, and reconfigurable designs that preserve connectivity for low-latency traffic. Opera combines deterministic reconfiguration with expander-based indirect forwarding and direct paths over time.

  • Static topologies: Static alternatives include Dragonfly, SlimFly, Diamond, WaveCube, and Quartz, which use localized bandwidth, optical interconnects, or multi-hop forwarding.These designs provide connected or sparse static networks without Opera’s time-varying topology.
  • Dynamic topologies: Helios, Mordia, and C-Through reactively establish high-bandwidth connections but rely on a separate packet-switched network for low-latency traffic.RotorNet uses deterministic reconfiguration and Valiant load balancing, but also requires a separate packet-switched low-latency network.
  • Dynamic topologies: ProjecToR maintains a connected base mesh while opportunistically reconfiguring free-space links in response to traffic changes.Its proposed weighted source-sink matching addresses concerns about a random base network’s support for skew.
  • Reconfigurable low-latency networks: OSA reserves circuit-switch ports to preserve low-latency connectivity during reconfiguration, while Megaswitch could potentially use a similar approach.These designs rely on multi-hop indirection for low-latency traffic.

8 Conclusions

Opera combines time-varying expander graphs for low-latency traffic with direct connections over time for high-throughput bulk traffic.

  • Opera implements time-varying expander graphs that support low-latency traffic while providing direct connections between endpoints over time.
  • 4× higher throughput for shuffle workloads and 60% higher supported load for skewed datacenter workloads were achieved versus cost-equivalent static networks.Short-flow completion times were not adversely impacted.

A Cost-normalization approach

The analysis compares cost-equivalent networks using Opera’s port-cost ratio α, with a three-tier folded Clos as the normalization basis and optical rotor switches as the likely cost-effective technology.

  • α is defined as Opera port cost divided by static network port cost for comparing cost-equivalent topologies.Opera ports include circuit-switch components, while static ports include a packet switch port, transceiver, and fiber.
  • For a folded Clos, α = 2(T −1)/F; for a static expander, α = u/(k−u).Here, T is the number of tiers, F is the oversubscription factor, u is the number of ToR uplinks, and k is the ToR radix.
  • A three-tier folded Clos normalizes comparisons while packet-switch radix k and host count H remain constant at each comparison point.The host count is derived from k, α, and the resulting oversubscription factor.
  • Optical rotor switches are considered likely most cost-effective because of approximately 3 dB attenuation and compatibility with single-mode or multimode signaling.

B Reducing cycle time at scale

Opera reduces cycle-time growth at larger scale by reconfiguring grouped circuit switches in parallel while preserving connectivity during reconfiguration.

  • Multiple circuit switches can reconfigure simultaneously, preventing cycle time from scaling quadratically with ToR radix.The remaining switches continue delivering a fully connected network during reconfiguration.
  • Opera port cost includes packet-switch, transceiver, fiber, and rotor-switch components, with rotor costs amortized across rotor ports.The cited table assumes 512-port rotor switches.
  • Grouping circuit switches and parallelizing their reconfiguration improves relative cycle time at larger scale.
  • A six-group parallelization increases cycle time 6× from k = 12 to k = 64, with a 90 MB bulk-flow cutoff at k = 64.The compared networks have 648 and 98,304 hosts, respectively.

C Additional scaling analysis

Additional scaling analysis finds nearly unchanged cost-normalized performance across network sizes, consistent with converging path lengths for Opera and static expanders.

  • Nearly identical performance at k = 12 and k = 24 indicates cost-normalized performance is nearly independent of scale across the evaluated topologies.
  • Average path lengths converge for larger networks, while worst-case path length is 4 ToR-to-ToR hops for all evaluated networks at k = 24 and above.Opera and static expanders were evaluated for k between 12 and 48 across cost points α.

D Spectral efficiency and path lengths

Opera’s topology slices achieve path lengths close to the best static expanders while supporting evaluation across network sizes, costs, and workloads. Path length is not strongly determined by spectral gap when host count is fixed.

  • Spectral efficiency: Spectral gap measures how closely a graph approaches an optimal Ramanujan expander, with larger gaps indicating better expansion.
  • Path lengths: Opera’s 108 topology slices achieve average path lengths close to the best achievable with static expanders.The comparison uses a 648-host, 108-rack network with k = 12-radix ToRs.
  • Spectral efficiency: Average and worst-case path length is not a strong function of spectral gap when the number of hosts is fixed.
  • Evaluation scope: Figure 15 reports throughput for hotrack, skew[0.2,1], and permutation workloads at k = 24 ports.
  • Evaluation scope: The evaluation covers path lengths from k = 12 networks with approximately 650 hosts to k = 48 networks with approximately 98,000 hosts under varying relative cost assumptions.

E Additional failure analysis

Opera recomputes routes around failures, with path lengths evaluated across failure conditions and topology slices. Its fault tolerance exceeds a 3:1 folded Clos but remains below a more heavily provisioned u = 7 expander.

  • Failure response: Opera recomputes paths around failed links, ToRs, and circuit switches, generally producing longer paths than under zero failures.
  • Failure response: Path-length analysis reports average and maximum values across all topology slices as failure degree increases.
  • Comparisons: Opera has better fault tolerance than the 3:1 folded Clos, while the u = 7 expander performs better still.The u = 7 expander has significantly more links and switches, plus higher fanout at each ToR.
  • Failure measurements: Figure 18 measures average and worst-case path lengths for a 108-rack Opera network with 6 circuit switches and k = 12-port ToRs under varied failures.Only finite-length paths are included, while disconnected ToR pairs are reported separately.
  • Comparisons: Figures 19 and 20 compare connectivity loss and path-length impact under link and ToR failures for a 3:1 folded Clos and a u = 7 expander.
Loading 1903.12307v1…