PaperlayerView exact revision on arXiv

Source-linked AI summary

An In-Depth Analysis of the Slingshot Interconnect

Daniele De Sensi, Salvatore Di Girolamo, Kim H. McMahon, Duncan Roweth, Torsten Hoefler

arXiv:2008.08886v1cs.DCcs.NIcs.PF

TL;DR

As systems scale and workloads become more data-centric, interconnect performance and tail latency matter across HPC, data centers, and AI/ML. SLINGSHOT combines high-radix Ethernet switching with adaptive routing, congestion control, and QoS. The paper reports that its applications are less affected by congestion than on previous-generation networks and demonstrates bandwidth guarantees across traffic classes.

  • Problem

    As systems scale and HPC, data-center, and AI/ML workloads become more data-centric, tail latency remains a concern for latency-sensitive applications.

    Source: §I · p. 1

  • Method

    SLINGSHOT combines high-radix Ethernet switches with adaptive routing, congestion control, QoS, and interoperability with standard Ethernet equipment.

    Source: §I · p. 1

  • Results

    Applications on SLINGSHOT are much less affected by congestion than on previous-generation networks, and allocation policies have a much lower impact on performance.

    Source: §V · p. 11

  • Takeaways & Limitations

    The paper demonstrates bandwidth guarantees for jobs running in separate traffic classes, a supported QoS capability for network workloads.

    Source: §V · p. 12

Abstract

from arXiv · show

The interconnect is one of the most critical components in large scale computing systems, and its impact on the performance of applications is going to increase with the system size. In this paper, we will describe Slingshot, an interconnection network for large scale computing systems. Slingshot is based on high-radix switches, which allow building exascale and hyperscale datacenters networks with at most three switch-to-switch hops. Moreover, Slingshot provides efficient adaptive routing and congestion control algorithms, and highly tunable traffic classes. Slingshot uses an optimized Ethernet protocol, which allows it to be interoperable with standard Ethernet devices while providing high performance to HPC applications. We analyze the extent to which Slingshot provides these features, evaluating it on microbenchmarks and on several applications from the datacenter and AI worlds, as well as on HPC applications. We find that applications running on Slingshot are less affected by congestion compared to previous generation networks.

I. INTRODUCTION

Interconnect performance becomes more consequential as systems scale, while growing data-centric workloads and tail-latency demands sharpen the need for networks compatible with Ethernet.

I. INTRODUCTION

  • Interconnect impact on system performance grows with scale, and HPCG’s roughly 50× lower performance than HPL partly reflects its greater communication intensity.The introduction uses this contrast to motivate efficient interconnects for exploiting system compute capacity.Source: §I · p. 1
  • Data-centric workloads span HPC, data centers, and AI/ML, while latency-sensitive applications make tail latency an important network concern.Web search is cited as needing 99th-percentile latencies of a few milliseconds; HPC performance can also depend strongly on message latency.Source: §I · p. 1
  • SLINGSHOT combines 64-port, 200 Gb/s switches with adaptive routing, congestion control, and QoS while interoperating with standard Ethernet equipment.Ports negotiate Ethernet features with attached devices; SLINGSHOT-specific additions are used for communication with SLINGSHOT switches and NICs.Source: §I · p. 1
  • The study evaluates network features with microbenchmarks and applications, using 100 Gb/s ConnectX-5 RoCE NICs and comparing congestion behavior with ARIES.ARIES is described as having a similar topology and routing algorithm; the introduction reports that SLINGSHOT is only marginally affected by network noise.Source: §I · pp. 1–2

II. SLINGSHOT ARCHITECTURE

The architecture centers on the ROSETTA switch and its tiled crossbar, with request/grant mechanisms and queue information supporting switching, QoS, and congestion management.

A. Switch Technology (ROSETTA)

  • ROSETTA provides 64 ports at 200 Gb/s per direction, and its 32 tiles implement switching, adaptive routing, and congestion management.The tiles form four rows of eight; each handles two ports, and the hierarchy avoids a 64-port arbiter. The ASIC also has 32 peripheral function blocks.Source: A. Switch Technology (ROSETTA) · p. 2
  • The authors report that 2200 Gb/s Ethernet NICs were unavailable at the time of writing.Source: A. Switch Technology (ROSETTA) · p. 2
  • A virtual output-queued design determines routes before sending data; output tiles grant transmission after receiving requests, helping avoid head-of-line blocking.Data remains in input buffers until the required resources are available; requests and grants are central to QoS management.Source: A. Switch Technology (ROSETTA) · p. 2
  • Request-queue credits estimate path congestion for adaptive routing, while end-to-end acknowledgments track outstanding packets for congestion control.Source: A. Switch Technology (ROSETTA) · p. 2
  • Physically separated crossbars keep different message types from interfering, so large transfers do not slow requests and grants.Source: A. Switch Technology (ROSETTA) · p. 2
  • 350 ns mean and median switch latency under RoCE, with the distribution between 300 and 400 ns except for a few outliers.The test uses standard RoCE NICs, so it does not exploit all features of SLINGSHOT’s specialized Ethernet protocol.Source: A. Switch Technology (ROSETTA) · p. 3, Figure 2 · p. 3

B. Topology

SLINGSHOT uses a low-diameter Dragonfly topology that scales to large systems; isolated tests show limited latency and bandwidth differences across node distances.

B. Topology

  • Dragonfly groups are fully connected, with switches fully connected within groups; the resulting topology has a diameter of three switch-to-switch hops.Each switch connects to 16 endpoints and uses its remaining 48 ports for inter-switch connectivity; within-group links are copper and inter-group links are optical.Source: B. Topology · p. 3, Figure 3 · p. 3
  • 40% worst-case allocation impact on 8B-message latency, under 10% latency difference from 16KiB, and less than 15% bandwidth difference across distances.The isolated-system measurements compare nodes one, two, or three inter-switch hops apart; same-switch ports on the same or different tiles showed no significant latency difference.Source: B. Topology · p. 3, Figure 4 · p. 3
  • The full-bandwidth construction describes 279,040 endpoints in 545 groups, while the addressing scheme limits practical systems to 261,632 nodes in 511 groups.Source: B. Topology · p. 3
  • The large endpoint count permits multiple connections from each computing node to the network, increasing injection bandwidth and improving resiliency to NIC failures.Source: B. Topology · p. 4

C. Routing

SLINGSHOT selects paths using estimated congestion and path length, while its hardware congestion control addresses endpoint contention that routing alone cannot avoid. It tracks endpoint pairs and throttles contributing streams to limit interference.

C. Routing

  • Adaptive routing estimates congestion on up to four minimal and non-minimal paths, then chooses using both congestion and path length.Queue depth estimates congestion; information is shared within a switch and between neighboring switches, with minimal paths favored to limit the cost of extra hops.Source: C. Routing · p. 4

D. Congestion Control

  • Endpoint congestion affects all paths between two nodes, so adaptive routing cannot route around receiver-side bottlenecks.In contrast, adaptive routing can avoid intermediate congestion by changing packet paths.Source: D. Congestion Control · p. 4
  • SLINGSHOT tracks in-flight packets by endpoint pair and applies fast backpressure to contributing sources, sparing unrelated streams from throttling.The authors say this frees buffers, avoids network-wide head-of-line blocking, and reduces tail latencies.Source: D. Congestion Control · p. 4

E. Quality of Service (QoS)

SLINGSHOT traffic classes provide configurable quality-of-service controls, while its Ethernet-compatible enhancements support communication with standard Ethernet devices and networks.

E. Quality of Service (QoS)

  • Traffic classes can be shared across jobs, so congestion control remains necessary within a class despite QoS isolation.Administrators can tune priority, ordering, bandwidth guarantees and limits, lossiness, and routing bias; minimum bandwidth guarantees must fit within available bandwidth.Source: E. Quality of Service (QoS) · p. 4

F. Ethernet Enhancements

  • SLINGSHOT mixes enhanced and standard Ethernet traffic at packet-level granularity and connects to third-party Ethernet devices and networks.Its enhancements include smaller minimum frames, Ethernet-header-free IP packets, and no inter-packet gap; it also provides FEC, link-level reliability, lane degradation, and end-to-end retry.Source: F. Ethernet Enhancements · p. 5

G. Software Stack

The software stack carries HPC traffic over RoCEv2 and supports libfabric-based communication libraries. On SHANDY, MPI_Alltoall exceeds 90% of theoretical peak bandwidth, with a reported drop for 256-byte messages.

G. Software Stack

  • HPC libraries use libfabric over RoCEv2, and Cray MPI is derived from MPICH with SLINGSHOT-specific optimizations.Any MPI implementation supporting libfabric can be used; for small messages, MPI adds only marginal overhead to libfabric.Source: G. Software Stack · p. 5, Figure 5 · p. 5
  • MPI_Alltoall bandwidth was measured across different process counts per node and message sizes on all 1,024 SHANDY nodes, with a performance drop observed for 256-byte messages.To reduce memory usage, the implementation switches algorithms for messages larger than 256 bytes.Source: Figure 6 · p. 6, G. Software Stack · p. 6

III. PERFORMANCE STUDY

  • The performance study examines congestion control and QoS on CRYSTAL, MALBEC, and SHANDY, including two SLINGSHOT systems at different scales.The systems were booked for exclusive use to control the environment and avoid interference from other users.Source: §III · p. 6

A. Congestion Control

Across the tested applications and microbenchmarks, SLINGSHOT is less affected by congestion than ARIES, including under varied allocations and system sizes. Its measured effects also remain limited under full-scale and bursty congestion tests.

A. Congestion Control

  • Tests divided nodes into victim and aggressor partitions, using incast or all-to-all patterns and varying split, allocation, and aggressor processes per node.Victims included applications and microbenchmarks; microbenchmark runs continued until at least 200 executions, four seconds, and a 95% confidence interval within 5% of the median.Source: A. Congestion Control · p. 6

TYPE APPL. DESCRIPTION

  • Under incast, Silo, Xapian, and Img-dnn degraded severely on ARIES but showed no relevant effect on SLINGSHOT; Sphinx degraded less on ARIES.The experiments used linear allocation and a 10%/90% victim/aggressor ratio; ARIES also showed higher tail latency, which congestion increased further.Source: TYPE APPL. DESCRIPTION · p. 7, Figure 8 · p. 7
  • SLINGSHOT's worst observed slowdown was 1.3x, versus a maximum 93x on ARIES; intermediate all-to-all congestion did not significantly affect either system.The authors attribute the limited intermediate-congestion impact to adaptive routing around congested links; the heatmap summarizes victim/aggressor combinations under linear allocation.Source: TYPE APPL. DESCRIPTION · p. 8, Figure 9 · p. 8
  • Across allocation strategies, SLINGSHOT congestion-impact values were below two in all but one case, with a less-spread distribution than ARIES.For ARIES, linear-allocation values stayed below 100, while interleaved and random allocations reached 150.Source: TYPE APPL. DESCRIPTION · p. 8, Figure 10 · p. 9
  • With aggressors using 24 processes per node, SLINGSHOT's maximum congestion impact was about 200 times lower than ARIES's.The comparison covers the higher-load case summarized in Figure 10.Source: TYPE APPL. DESCRIPTION · p. 8, Figure 10 · p. 9
  • At 128 nodes, maximum congestion impact fell from 154 to 40 on ARIES and from 2.3 to 1.5 on SLINGSHOT, compared with 512-node tests.The authors cite lower aggressor traffic and a higher fraction of available global bandwidth as possible explanations for the ARIES change.Source: TYPE APPL. DESCRIPTION · p. 9, Figure 10 · p. 9
  • On 1,024 SHANDY nodes, LAMMPS showed at most 3.55x slowdown when 75% of nodes were assigned to the incast aggressor.MILC and HPCG results at a 25%/75% aggressor/victim ratio are absent because those applications require power-of-two node counts.Source: TYPE APPL. DESCRIPTION · p. 9, Figure 11 · p. 9
  • For bursty incast, medium-size messages produced congestion impact up to 1.21, while small or large messages did not affect the victim.Large messages triggered throttling, whereas medium messages could build congestion before control reacted; large bursts and short gaps produced the highest impact, and bursts of 106 messages matched persistent congestion.Source: TYPE APPL. DESCRIPTION · p. 9, Figure 12 · p. 9

B. Traffic Classes

Traffic classes can separate jobs and shape bandwidth allocation, while congestion control addresses congestion within and across classes. In the reported tests, a separate class reduced MPI_Allreduce’s slowdown, and configured minimum bandwidths were enforced when two jobs used separate classes.

B. Traffic Classes

  • Traffic classes can protect a job or allocate resources fairly or unfairly, while congestion control addresses congestion within and across classes.The paper distinguishes the two as orthogonal concepts; fair resource assignment does not prevent congestion caused by buffer-filling jobs.Source: B. Traffic Classes · pp. 9–10
  • 2.85x slowdown in the same traffic class fell to 1.15x in a separate class for the 8B MPI_Allreduce co-running with a 256KiB MPI_Alltoall.The tests used MALBEC with bandwidth tapered to 25%; the slowdown is relative to isolated execution.Source: Figure 13 · p. 10, B. Traffic Classes · p. 10
  • When both bisection jobs shared TC1, they fairly shared available bandwidth after the second job started, and the remaining job used all bandwidth after the first ended.Figure 14 contrasts this shared-class behavior with the separate-class configuration.Source: B. Traffic Classes · p. 10, Figure 14 · p. 10
  • With separate classes, the first bisection job retained its 80% minimum and the second received 20%, including 10% of unallocated bandwidth.After the first job terminated, the second used all available bandwidth.Source: B. Traffic Classes · p. 10, Figure 14 · p. 10

A. Interconnection Networks

Existing interconnects span different topologies and features, while low-diameter networks face deployment and routing challenges. The paper presents Slingshot’s congestion control and network features as addressing those challenges.

A. Interconnection Networks

  • InfiniBand is an open high-performance networking standard with common fat-tree implementations and features including adaptive routing, congestion control, and traffic classes.Large production systems usually avoid congestion control because tuning the algorithm is difficult; Ethernet interoperability requires dedicated gateways.Source: A. Interconnection Networks · pp. 10–11
  • ARIES uses Dragonfly topology and fewer optical links than fat trees, while TofuD uses a 6D mesh/torus and dynamic packet slicing for transmission performance or link-failure tolerance.TofuD also provides an offload engine for collective operations.Source: A. Interconnection Networks · p. 11
  • Dragonfly+ connects switches within each group through a fat tree and uses adaptive routing that re-evaluates paths at each hop, with congestion notifications between switches.The described implementation uses Mellanox InfiniBand hardware.Source: A. Interconnection Networks · p. 11
  • Low-diameter networks have seen limited deployment because of difficult routing, and classical ECMP is ineffective when minimal paths are scarce; Slingshot pairs low diameter with effective congestion control.The paper frames this as a stepping stone toward HPC datacenters.Source: A. Interconnection Networks · p. 11
  • Slingshot combines end-to-end congestion control, flexible traffic classes, native Ethernet interoperability, and adaptive routing that supports high utilization for in-order RoCE traffic.The congestion-control algorithm is described as quickly reacting to congestion and remaining stable across applications and microbenchmarks.Source: A. Interconnection Networks · p. 11

B. Interconnection Networks Benchmarking

The study evaluates Slingshot under isolated and congested conditions across microbenchmarks and real applications. It broadens congestion evaluation beyond GPCNet’s small set of victim benchmarks and fixed experimental settings.

B. Interconnection Networks Benchmarking

  • The evaluation measures Slingshot’s isolated performance and performance under congestion across microbenchmarks and real applications.The paper describes this as its first extensive evaluation across those workload types.Source: B. Interconnection Networks Benchmarking · p. 11
  • Unlike GPCNet’s random-ring and MPI_Allreduce victim microbenchmarks, this study uses varied microbenchmarks and HPC and datacenter applications to examine congestion impact.GPCNet’s stated goal was portable congestion benchmarking with a small set of victim microbenchmarks.Source: B. Interconnection Networks Benchmarking · p. 11
  • GPCNet fixes victim message size, allocation, and aggressor/victim ratio, whereas the study varies these factors because each affects observed congestion.Source: B. Interconnection Networks Benchmarking · p. 11

V. CONCLUSIONS

The paper describes Slingshot’s Ethernet switches, adaptive routing, congestion control, and QoS management, then evaluates workloads in isolation and under concurrency. It reports lower congestion impact than previous-generation networks and bandwidth guarantees for jobs in separate traffic classes.

V. CONCLUSIONS

  • The evaluated network features include high-radix Ethernet switches, adaptive routing, congestion control, and QoS management.The paper evaluates performance both in isolation and with concurrent workloads.Source: §V · p. 11
  • Applications on Slingshot are less affected by congestion than on previous-generation networks, and its congestion control works across microbenchmarks and HPC and datacenter applications.The paper also reports that allocation policies have a much lower performance impact than on previous-generation networks.Source: §V · p. 11
  • Slingshot can provide bandwidth guarantees to jobs running in separate traffic classes.Source: §V · p. 12
  • The paper’s information is intended to help HPC and datacenter operators, administrators, users, and programmers optimize, deploy, and manage parallel applications.The authors describe understanding interconnect features as a prerequisite for optimizing operations and resource utilization in clouds and datacenters.Source: §V · p. 12
Loading 2008.08886v1…