Source-linked AI summary
Jellyfish: Networking Data Centers Randomly
Ankit Singla, Chi-Yao Hong, Lucian Popa, P. Brighten Godfrey
TL;DR
Existing high-bandwidth data-center networks have rigid structures that hinder incremental expansion. Jellyfish uses a degree-bounded random graph to enable flexible growth, and supports more servers than same-equipment fat-trees while maintaining at least as high bandwidth. The paper also examines routing, physical construction, and cabling challenges, including higher inter-container cabling costs at massive scale.
Problem
Existing high-bandwidth data-center network designs have rigid structures that interfere with incremental expansion.
Method
Jellyfish constructs a degree-bounded random graph among top-of-rack switches and expands it through random edge swaps.
Results
Jellyfish supports 25% more servers than a same-equipment fat-tree while providing at least as high bandwidth, with the advantage increasing with network size and switch port-count.
Takeaways & Limitations
Random graphs provide a flexible architecture for incremental and heterogeneous expansion while enabling high capacity, short paths, and resilience to failures and miswirings.
Takeaways & Limitations
At massive scale, many Jellyfish cables are inter-container links requiring expensive optical connectors, which can produce excessive cabling costs compared with a fat-tree.
Abstract
from arXiv · showhide
Industry experience indicates that the ability to incrementally expand data centers is essential. However, existing high-bandwidth network designs have rigid structure that interferes with incremental expansion. We present Jellyfish, a high-capacity network interconnect, which, by adopting a random graph topology, yields itself naturally to incremental expansion. Somewhat surprisingly, Jellyfish is more cost-efficient than a fat-tree: A Jellyfish interconnect built using the same equipment as a fat-tree, supports as many as 25% more servers at full capacity at the scale of a few thousand nodes, and this advantage improves with scale. Jellyfish also allows great flexibility in building networks with different degrees of oversubscription. However, Jellyfish's unstructured design brings new challenges in routing, physical layout, and wiring. We describe and evaluate approaches that resolve these challenges effectively, indicating that Jellyfish could be deployed in today's data centers.
1 Introduction
Existing high-bandwidth data-center networks are poorly suited to incremental growth because rigid structures force coarse expansion or compromises in bandwidth and cost. Jellyfish uses a random-graph interconnect to support flexible expansion while retaining high capacity.
- Incremental expansion adds servers and network capacity as data centers grow or deploy more bandwidth-hungry applications.
- Fat-trees permit full-bisection designs only at coarse sizes determined by available switch port counts.Workarounds include replacing switches, uneven oversubscription, or reserving unused ports.
- Jellyfish is a degree-bounded random graph among top-of-rack switches that supports arbitrary-size, heterogeneous, incrementally expandable networks.New racks or switches can be incorporated through a few random edge swaps.
- Routing, physical construction, and cabling are challenges created by Jellyfish’s lack of regular structure, but the paper describes approaches suggesting deployment in today’s data centers.
- Jellyfish grows incrementally at only 40% of the expense of prior incremental Clos-network expansion work.
- 25% more servers are supported than by a same-equipment fat-tree while providing at least as high bandwidth, with the advantage increasing with network size and switch port-counts.
2 Related Work
Prior high-capacity proposals generally rely on rigid structure or planned expansion, leaving incremental growth insufficiently addressed. Jellyfish instead applies random graphs to expansion and quantifies their efficiency against traditional topologies.
- Folded-Clos, server-forwarding, optical, and expander-graph proposals exploit special structure for topology or routing.
- Many prior architectures require extensive replacement or rewiring, coarse expansion, or reserved server ports to grow while preserving their intended structure.
- Scafida and SWDC use randomness but retain edge correlations or lattice structure, and neither investigates whether its characteristics survive incremental expansion.
- Jellyfish improves on fat-trees in bisection bandwidth and diameter, and has higher bandwidth than same-equipment SWDC topologies.
- LEGUP optimizes upgrades within Clos structure but may require substantial topology overhauls unless free ports are reserved; Jellyfish targets almost any desirable scale.
- The paper applies random graphs to incremental expansion and quantifies their efficiency gains over traditional data-center topologies.
3 Jellyfish Topology
Jellyfish constructs a degree-bounded random graph at the top-of-rack switch layer, using random links and swaps to achieve flexible expansion. Its random connectivity is intended to provide short paths and high throughput.
- Each top-of-rack switch divides its ports between links to other switches and attached servers; equal-degree networks form RRG(N, k, r).With N racks, the regular case supports N(k − r) servers.
- A random-pair procedure connects switches with free ports while avoiding duplicate neighbors, producing sufficiently uniform graphs with desired empirical performance.
- A newly added switch can be incorporated by removing one existing link and replacing it with two links to the new switch, leaving at most one unmatched port overall.
- The random graph’s fluid capacity supports arbitrary numbers of switches, heterogeneous degree distributions, and new switches through a few random link swaps.
- Low average path length reduces network capacity consumed per byte, allowing more flows at high throughput when routing uses full capacity.
- >99.5% of source-destination pairs in a 686-server Jellyfish are reachable in fewer than 6 hops, versus 7.5% for the corresponding fat-tree.
4 Jellyfish Topology Properties
Jellyfish combines high capacity, short paths, resilience, and flexible incremental expansion through a degree-bounded random-graph topology. Across comparisons, it supports more servers or bandwidth than fat-trees and remains close to carefully optimized degree-diameter graphs.
- Efficiency: At the same cost as a 16,000-server fat-tree, Jellyfish supports more than 20,000 servers at full bisection bandwidth.Jellyfish also permits trading bisection bandwidth for more servers or fewer switches.
- Efficiency: Jellyfish supports 27% more servers at full capacity than a same-equipment fat-tree below 900 servers, with the advantage improving at larger scales.The comparison uses random-permutation traffic and optimal routing; the largest evaluated size is 874 servers.
- Efficiency: Jellyfish achieves approximately 91% or more of the throughput of best-known degree-diameter graphs, which serve as capacity benchmarks rather than incrementally expandable designs.The optimized graphs outperform Jellyfish, but the measured gap is small in the tested comparisons.
- Flexibility: Jellyfish’s expansion and topology design support continuous sizing, heterogeneous hardware, and cost-efficient capacity growth compared with Clos-based alternatives.Its lower-bound normalized bisection bandwidth is independent of network size, although average path length increases slowly as networks grow.
- Flexibility: Incremental expansion can add one rack or switch with rewiring limited to the added ports, while preserving high bandwidth and short paths.Incrementally built topologies closely match those constructed from scratch in capacity and path length, and heterogeneous switch port-counts are supported.
5 Routing & Congestion Control
Jellyfish requires routing beyond standard ECMP to realize its capacity, but k-shortest paths with MPTCP approach optimal throughput and preserve a substantial server-count advantage over same-equipment fat-trees.
- Routing and Congestion Control: ECMP provides insufficient path diversity for Jellyfish, motivating k-shortest-path routing that can use longer-than-shortest paths.The evaluation compares ECMP with k-shortest paths using an MPTCP-based simulator.
- Fat-tree Throughput Comparison: With the same equipment as a 686-server fat-tree, Jellyfish supports 780 servers with TCP and 805 with MPTCP at the same or higher performance.Under ideal routing and congestion control, the corresponding figure is 874 servers.
- Routing and Congestion Control: 86%−90% of optimal throughput is achieved by simple k-shortest-path forwarding with MPTCP across the evaluated cases.Results are averaged over 10 runs.
- Fat-tree Throughput Comparison: Jellyfish supports 3,330 servers versus 2,662 for the fat-tree at the largest simulated scale, a >25% improvement after routing and congestion-control inefficiencies.At smaller scale, Jellyfish supports 496 servers versus 432, an improvement of approximately 15%.
- Fairness: Jellyfish and fat-tree achieve similarly good flow fairness, with Jain’s fairness index near 99% for both topologies.Jellyfish has more flows because it supports more servers in the same-equipment comparison.
- Practical Routing: Practical routing approaches include precomputed k-shortest paths implemented through OpenFlow, SPAIN, or MPLS mechanisms.Each switch maintains routing information for k shortest paths to every other switch.
6 Physical Construction and Cabling
Jellyfish’s irregular topology creates wiring challenges, but layout optimization, localized cabling, automated blueprints, and limited rewiring provide practical construction strategies across small and massive deployments.
- Wiring Challenges: Cabling complexity is a central concern because irregular layouts may be difficult to wire and more susceptible to wiring errors.The design also seeks cable bundles that reduce manual wiring effort and expense.
- Wiring Challenges: Jellyfish cabling can be planned with an automatically generated blueprint based on the topology and physical data-center layout.Workers then manually connect cables according to the blueprint.
- Wiring Errors: Fixing 10% miswirings is argued to be relatively inexpensive because cabling labor is estimated at approximately 10% of total cabling cost.The paper presents this as a pessimistic cost estimate under a total cabling cost of 50% of network cost.
- Small Clusters and CDCs: Using fewer switches for the same server pool lets Jellyfish require 15−20% fewer cables than a fat-tree, depending on scale.The resulting savings also create more room and budget for additional servers.
- Small Clusters and CDCs: A random-connect patch panel could encode Jellyfish’s interconnect internally while presenting workers with a regular, easier-to-wire external pattern.The paper suggests this could accelerate manual cabling in container data centers.
- Cabling under Expansion: Jellyfish expansion requires moving two cables for every two added network ports, with affected connections automatically identified.In a switch-cluster configuration, the work is concentrated in one location or between clusters.
- Massive-Scale Data Centers: At massive scale, inter-container cables may require expensive optical connectors and cause excessive cabling costs compared with a fat-tree.The paper investigates localization to reduce this cost while preserving capacity.
- Massive-Scale Data Centers: Localizing 60% of Jellyfish’s network connections reduces throughput by less than 6% relative to unrestricted Jellyfish.The equivalent fat-tree has 53.6% local links, while Jellyfish remains higher-capacity in the reported comparison.
7 Conclusion
The paper concludes that random-graph data-center networks provide a flexible alternative to structured designs, combining expansion flexibility with high capacity, short paths, and resilience.
- Conclusion: Random graphs are presented as a flexible architecture for data-center networks.The conclusion emphasizes their suitability for incremental and heterogeneous expansion.
- Conclusion: Jellyfish addresses incremental and heterogeneous expansion while enabling high capacity, short paths, and resilience to failures and miswirings.These properties summarize the paper’s supported conclusion about random-graph interconnects.