PaperlayerView exact revision on arXiv

Source-linked AI summary

TopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed Training Jobs

Weiyang Wang, Moein Khazraee, Zhizhen Zhong, Manya Ghobadi, Zhihao Jia, Dheevatsa Mudigere, Ying Zhang, Anthony Kewitsch

arXiv:2202.00433v3cs.NIcs.DC

TL;DR

Fat-tree traffic assumptions do not fit DNN jobs well, whose communication patterns vary with parallelization, while jointly searching parallelization and network topology is difficult. TOPOPT co-optimizes them and reduces training iteration time by up to 3.4× versus a similar-cost Fat-tree.

  • Problem

    Fat-tree traffic assumptions poorly fit DNN jobs, whose communication patterns vary with parallelization and can consume up to 60% of iteration time.

    Source: §1 · p. 1, §2.2 · p. 3

  • Method

    TOPOPT alternates between compute×communication and communication×topology searches to jointly select a DNN job’s parallelization strategy, network topology, and routing.

    Source: §1 · p. 1, §4.1 · p. 5

  • Results

    Up to 3.4× reduction in training iteration time versus a similar-cost Fat-tree across six representative DNN models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper demonstrates an optical-device direct-connect system that jointly optimizes DNN parallelization and topology to accelerate training.

    Source: §9 · p. 14

    TOPOPT assumes traffic patterns remain unchanged between iterations, which may not hold for GNN or MoE models.

    Source: §7 · p. 14

Abstract

from arXiv · show

We propose TopoOpt, a novel direct-connect fabric for deep neural network (DNN) training workloads. TopoOpt co-optimizes the distributed training process across three dimensions: computation, communication, and network topology. We demonstrate the mutability of AllReduce traffic, and leverage this property to construct efficient network topologies for DNN training jobs. TopoOpt then uses an alternating optimization technique and a group theory-inspired algorithm called TotientPerms to find the best network topology and routing plan, together with a parallelization strategy. We build a fully functional 12-node direct-connect prototype with remote direct memory access (RDMA) forwarding at 100 Gbps. Large-scale simulations on real distributed training models show that, compared to similar-cost Fat-Tree interconnects, TopoOpt reduces DNN training time by up to 3.4x.

1 Introduction

TopoOpt addresses growing DNN training demands and Fat-tree bottlenecks by jointly optimizing physical network topology and parallelization strategy for training jobs.

1 Introduction

  • Fat-tree networks are becoming a bottleneck for distributed DNN training, while prior work co-optimizes computation and communication without optimizing physical topology [69] [77] [85].Fat-tree fabrics were designed for unpredictable legacy workloads dominated by short transfers.Source: §1 · p. 1
  • Production-cluster analysis shows that communication overhead rises dramatically with worker count and that a DNN job’s traffic pattern depends on its parallelization strategy.Source: §1 · p. 1
  • TOPOPT creates dedicated job partitions using reconfigurable optical switches and patch panels, jointly optimizing network topology and parallelization strategy within each partition.Source: §1 · p. 1
  • An offline alternating optimization framework updates parallelization under a fixed topology, then uses the resulting traffic demand to find a topology and feeds it back until convergence.TotientPerms exploits the mutability of AllReduce transfers to construct permutations that carry AllReduce efficiently while also supporting low-hop Model Parallel transfers.Source: §1 · pp. 1–2
  • TopoOpt reduces training iteration time by up to 3.4× versus a similar-cost Fat-tree and is on average 3.2× cheaper than an ideal full-bisection-bandwidth Fat-tree.Evaluations covered six representative DNN models.Source: §1 · p. 2

2 Motivation

Production DNN training jobs are long-lasting, and their traffic patterns repeat across iterations, creating an opportunity to precompute traffic for demand-aware networks. Measurements also show network communication can consume a substantial share of training time.

2.1 Background on Distributed DNN training

  • DNN training generates MP transfers for activations and gradients and AllReduce transfers for weight synchronization; hybrid data and model parallelism can produce both across accelerators.Depending on parallelization, these dependencies may instead result in local memory accesses.Source: §2.1 · p. 2
  • For the illustrated DLRM, hybrid parallelism reduces the maximum transfer size from 44 GB to 4 GB compared with pure data parallelism.The hybrid pattern includes 4 GB AllReduce transfers and 32 MB MP transfers.Source: §2.1 · p. 3, Figure 1 · p. 3

2.2 Production Measurements

  • Most jobs use 32 to 700 workers and last over 10 hours; the top 10% take more than 96 hours to finish.The passage describes production jobs at Meta as lasting up to weeks.Source: §2.2 · p. 3, Figure 2 · p. 3
  • Network overhead reaches 60% of a DNN training iteration in Meta’s production environment as GPU counts increase.The measurements cover six production jobs as GPU counts rise from 8 to 128.Source: §2.2 · p. 3, Figure 3 · p. 4
  • Production jobs show periodic, predictable traffic: their per-iteration heatmaps remain identical throughout training, while MP traffic varies with model and placement.Traffic patterns still change within an iteration across forward, backward, and AllReduce phases.Source: §2.2 · p. 3, Figure 4 · p. 4

3 TOPOOPT System Design

TOPOOPT is a demand-aware optical direct-connect system that jointly optimizes DNN parallelization and network topology. It dedicates a server shard to each job and configures that shard to realize its selected topology.

  • Each server connects to a core layer of optical switches, which partition the cluster into job-specific shards sized to each job’s server request.TOPOOPT finds the parallelization strategy and topology offline, then reconfigures the switches before training.Source: §3 · p. 4
  • In the testbed, one 100 Gbps NIC with 4×25 Gbps interfaces provides each server with degree four.Server degree d is the number of interfaces and typically matches the number of installed NICs; breakout cables or co-packaged optical NVLinks can increase it when NICs are limited.Source: §3 · p. 4
  • Host-based forwarding lets servers relay traffic to destinations without direct links, avoiding blocked traffic when server degree is below the number of communicating neighbors.Hosts act as switches and forward incoming traffic toward its destination.Source: §3 · p. 4
  • TOPOOPT uses one-shot reconfiguration because current OCS latency is too high for within-iteration changes, keeping each job’s topology fixed during training.Patch panels suit jobs whose topology remains intact throughout training; OCSs could support within-iteration reconfiguration if their latency were adequate.Source: §3 · p. 4

4 Co-optimizing Parallelization Strategy and Network Topology

TOPOOPT co-optimizes parallelization strategy, communication, network topology, and routing through an alternating search. It exploits AllReduce traffic mutability to build topologies that serve AllReduce traffic while reducing hop-counts for model-parallel transfers.

4.1 Alternating Optimization

  • Alternating optimization searches Comp.×Comm. and Comm.×Topo. planes in turn, feeding improved parallelization and device placement into topology-and-routing search until convergence or k iterations.The Comp.×Comm. plane uses FlexFlow’s MCMC search; the Comm.×Topo. plane uses TOPOLOGYFINDER.Source: §4.1 · p. 5

4.2 TOPOLOGYFINDER Algorithm

  • TOPOLOGYFINDER allocates degree between AllReduce and model-parallel sub-topologies by traffic share, builds AllReduce links, then uses maximum-weight matching to add model-parallel links.It combines the sub-topologies, routes AllReduce with a modified coin-change algorithm, and uses k-shortest paths for model-parallel transfers.Source: §4.2 · pp. 5–6

4.3 Traffic Mutability and AllReduce Topology

  • AllReduce transfers are mutable without changing parallelization or device placement, unlike model-parallel transfers, enabling multiple ring orders to reduce model-parallel hop-counts.TOPOPT overlaps multiple permutations to serve AllReduce traffic; its DLRM example combines three rings to balance traffic while providing shorter model-parallel paths.Source: §4.3 · pp. 6–7
  • TotientPerms generates regular ring-AllReduce permutations from steps coprime with the group size; restricting steps to primes reduces the search space to O(n/ln n).For 12 servers, steps 1, 5, 7, and 11 yield four distinct ring permutations.Source: §4.3 · p. 7
  • SelectPermutations chooses regular AllReduce rings near a geometric sequence, bounding the AllReduce sub-topology diameter to O(dA·n^1/dA) under stated assumptions.The geometric hop distances create a topology similar to Chord.Source: §4.3 · p. 7

5 Large Scale Simulations

The section evaluates TopoOpt through simulations of dedicated and shared DNN clusters, cost comparisons, and sensitivity analyses of all-to-all traffic, host-based forwarding, and reconfiguration latency. It reports performance gains over Fat-tree in several workloads, while identifying conditions where forwarding or reconfiguration affects performance.

5.1 Methodology & Setup

  • FlexNet jointly searches topology and routing with parallelization, while FlexNetPacket simulates packet-level network behavior and multiple jobs sharing the fabric.FlexNet augments FlexFlow for network-aware evaluation; FlexNetPacket is event-based and built on htsim.Source: §5.1 · p. 8

5.2 Cost Analysis

  • TOPOPT’s cost overlaps with similar-cost Fat-tree, while an Ideal Switch costs 3.2× as much on average and OCS-based TOPOPT costs 1.33× as much as patch-panel TOPOPT.The section reports that SiP-ML and Expander are the most and least expensive fabrics, respectively.Source: Figure 10 · p. 9, §5.2 · p. 9

5.3 Performance Comparison on Dedicated Clusters

  • Compared with Fat-tree, TOPOPT improves iteration time by 2.8× for CANDLE and DLRM, 2.8× for VGG, 3× for BERT, and 2.1× for NCF.For DLRM and NCF, Ideal Switch further improves iteration time over TOPOPT by 1.3× and 1.7×, respectively; TOPOPT’s performance diverges from Ideal Switch particularly for NCF, which has many-to-many MP transfers.Source: §5.3 · p. 9, Figure 11 · p. 10

5.4 Impact of All-to-all Traffic

  • With batch size 128 and d = 4, TOPOPT matches Ideal Switch while Fat-tree is 2.7× slower; at batch size 2048, TOPOPT is 1.1× slower than Fat-tree.The latter case has all-to-all traffic at 80% of AllReduce traffic, and increasing server degree mitigates the performance problem.Source: §5.4 · p. 10, Figure 12 · p. 11

5.5 Impact of Host-based Forwarding

  • At d = 4, host-based forwarding averages 5.7 hops, yielding at least 5.7× overhead relative to Ideal Switch for all-to-all traffic; at d = 8, average path length falls to 3.At batch size 64, bandwidth tax is 1.11 at d = 4 and 1.05 at d = 8; at batch size 2048 and d = 4, it is 3.03. For batch size 128, the least-loaded link carries 39% less traffic than the most-loaded link at d = 4, and 59% less at d = 8.Source: §5.5 · p. 11

5.6 Performance on Shared Clusters

  • At full cluster load, TOPOPT’s tail iteration time is 3.4× faster than Fat-tree; averaged across loads, it is 3× and 1.4× faster than Fat-tree and Oversub. Fat-tree, respectively.TOPOPT also improves average iteration time by 1.7× and 1.15× versus Fat-tree and Oversub. Fat-tree, respectively; these gains are in a 432-server cluster with d = 8 and B = 100 Gbps.Source: §5.6 · pp. 11–12, Figure 16 · p. 12

5.7 Impact of Reconfiguration Latency

  • For BERT, enabling host-based forwarding makes OCS-reconfig iteration time 1.4× higher than without forwarding, whereas DLRM performs better with forwarding.At 1 µs reconfiguration latency, OCS-reconfig without forwarding matches TOPOPT; with forwarding, inaccurate demand estimates remain a problem.Source: Figure 17 · p. 12, §5.7 · p. 12

6 Prototype

This section describes a functional 12-node TopoOpt prototype, the changes needed to support its topology-aware communication, and its training performance against switch-based baselines.

Testbed setup

  • The prototype uses 12 servers with A100 GPUs and 100 Gbps NICs, configured for degree d = 4 and B = 25 Gbps, and compares against 100 Gbps and 25 Gbps switch baselines.RoCEv2 with DCB and PFC supports a lossless fabric for RDMA.Source: §6 · p. 12, Figure 18 · p. 13

Distributed training framework

  • FlexFlow trains ResNet50, BERT, VGG16, and CANDLE, while DLRM uses Facebook’s implementation; models and batch sizes are smaller than in simulation because the prototype is smaller.The testbed is an order of magnitude smaller than the simulation setup.Source: §6 · p. 12

Modifications to NCCL

  • TopoOpt modifies NCCL to respect its topology and routing preferences, integrating TotientPerms AllReduce permutations to load-balance parameter synchronization across ring-AllReduce permutations.NCCL’s default assumption that all network interfaces are mutually routable does not fit TopoOpt’s routing strategy.Source: §6 · p. 12

RDMA forwarding

  • To enable RDMA forwarding, TopoOpt uses NIC network partitioning to separate host-based forwarding from direct traffic and route forwarded RoCEv2 packets.The solution uses the Linux kernel for routing; NIC firmware and driver updates can instead route forwarded packets while bypassing the kernel.Source: §6 · pp. 12–13

Training performance

  • 2.0× faster time-to-accuracy to 90% than Switch 25Gbps; TopoOpt performs similarly to Switch 100Gbps in throughput and VGG19 time-to-accuracy.Figure 19 reports throughput across models, while Figure 20 shows the VGG19 ImageNet time-to-accuracy comparison.Source: §6 · p. 13, Figure 19 · p. 13, Figure 20 · p. 13

Impact of all-to-all traffic

  • 1.6× better training iteration time than Switch 25Gbps at batch size 512, where all-to-all traffic is 78% of AllReduce traffic.The test measures average iteration time over 320 DLRM iterations and increases embedding dimensions by 128× to create worst-case traffic.Source: §6 · p. 13, Figure 21 · p. 13

7 Discussion

TOPOPT targets long-running DNN training jobs with stable server assignments and traffic patterns, using a dedicated training fabric. The discussion describes adaptations for failures, elasticity, and multi-tenancy while identifying changing traffic patterns and dynamic joint optimization as open challenges.

Target workload

  • TOPOPT targets large DNN training jobs with hybrid data/model parallelism or data parallelism, fixed server assignments, and GPUs not shared across jobs.The intended workload is a set of long-lasting jobs whose participating servers do not change dynamically.Source: §7 · p. 13

Storage and control plane traffic

  • TOPOPT partitions training traffic over GPU NICs, while CPU NICs use a separate fabric for storage and other control-plane traffic.The passage describes Meta servers with eight GPUs, eight dedicated GPU NICs, and four additional CPU NICs; NVIDIA is noted as having similar architectures.Source: §7 · p. 13

Supporting dynamic scheduling and elasticity

  • For elastic jobs, OCSs or other fast-reconfigurable optical switches can quickly change participating servers, but jointly re-optimizing topology and parallelization adds an optimization dimension.The authors leave this more challenging dynamic optimization problem to future work.Source: §7 · p. 13

Handling failures

  • A single link failure does not disconnect TOPOPT’s graph, but can make an AllReduce ring inefficient; temporary MP links or port swaps recover connections, while fast optical reconfiguration can restore ring efficiency.TOPOPT can temporarily use an MP-dedicated link to recover a ring after fiber failure, and swap ports for permanent failures.Source: §7 · pp. 13–14

Supporting multi-tenancy

  • TOPOPT can support multi-tenancy by using NVIDIA MIG to represent one physical server as multiple logical servers in the topology.The discussion situates this capability in the context of multi-tenancy and NVIDIA MIG.Source: §7 · p. 13

TotientPerms in Fat-trees

  • TotientPerms may also help Fat-tree interconnects by load-balancing AllReduce traffic across multiple permutations to reduce network congestion.The technique is described as well-suited to reconfigurable optical interconnects, with possible independent interest for Fat-trees.Source: §7 · pp. 13–14

TOPOPT’s limitations

  • TOPOPT assumes traffic patterns remain stable between iterations, an assumption that may fail for GNN and MoE models; demand-aware optimization within iterations remains open work.Addressing this limitation would require fast-reconfigurable optical switches and a more sophisticated scheduling algorithm.Source: §7 · p. 14

8 Related Work

Related work on DNN training, parallelization, infrastructure, and optical networking generally assumes or optimizes around fixed or conventional network topologies. TOPOOPT is distinguished by jointly optimizing the physical topology and parallelization strategy, while scheduling methods are complementary.

Optimizing DNN training

  • Existing training frameworks distribute data or models but do not optimize physical topology; gradient compression targets data parallelism, while TOPOOPT co-optimizes topology and parallelization [69] [77] [79] [85].Blink requires a physical topology to construct spanning trees, and Zhao et al. study collective topologies but not general model-parallel traffic.Source: §8 · p. 14

DNN parallelization strategies

  • Automated parallelization frameworks such as FlexFlow [85], ColocRL, and MERLIN search strategy spaces but rely on conventional Fat-tree networks, whereas TOPOOPT jointly optimizes topology and strategy.Data and model parallelism are widely used by DNN frameworks including TensorFlow [44], PyTorch, and MXNet.Source: §8 · p. 14

DNN training infrastructures and schedulers

  • Existing training infrastructures assume nonreconfigurable, traffic-oblivious topologies, while TOPOOPT uses commodity reconfigurable interconnects; GPU-cluster scheduling methods are complementary and can apply to TOPOOPT.Examples include DGX SuperPOD, TPU clusters, supercomputers, and schedulers such as Gandiva, Themis [98], Tiresias, BytePS, and Pollux.Source: §8 · p. 14

Optical Interconnects

  • Optically reconfigurable and traffic-oblivious interconnects yield suboptimal topologies for repetitive DNN traffic, while hybrid electrical/optical clusters face cost and routing challenges [99].Hybrid designs can route AllReduce over optical links and model-parallel flows over electrical Fat-trees, but encounter TCP ramp-up, segregated routing, and cluster-partitioning uncertainty,,.Source: §8 · p. 14

9 Conclusion · A Tree-AllReduce and Other AllReduce Permutations

TOPOOPT jointly optimizes DNN parallelization strategy and topology, and the section shows how relabeling can extend AllReduce traffic manipulation from rings to tree-based algorithms. In particular, double binary trees and other isomorphic AllReduce graphs can support balanced communication under different node labelings.

9 Conclusion

  • TOPOOPT achieves up to 3.4× faster training iteration time than Fat-tree.The system uses optical devices to jointly optimize DNN parallelization strategy and topology.Source: §9 · p. 14
  • An alternating optimization algorithm explores the large space of Computation × Communication × Topology strategies for a DNN workload.TOPOOPT jointly optimizes the DNN parallelization strategy and topology to accelerate training jobs.Source: §9 · p. 14

A Tree-AllReduce and Other AllReduce Permutations

  • Tree-AllReduce reduces to a root with recursive halving, then broadcasts to the cluster with recursive doubling.In this algorithm, servers are logically connected as a tree.Source: A Tree-AllReduce and Other AllReduce… · p. 23
  • Double binary tree (DBT) uses two balanced binary trees with flipped leaf and in-tree labels to give nodes equal communication requirements and bandwidth-optimal operation [120].In the first tree, half the nodes are leaves and half are in-tree nodes; the second tree flips those labels.Source: A Tree-AllReduce and Other AllReduce… · p. 23
  • Permuting node labels for a pair of DBTs creates other tree pairs that perform AllReduce at the same speed, with cyclic modular permutations offering a way to limit cases.The passage illustrates alternative trees and corresponding traffic demand matrices for DLRM and CANDLE examples.Source: A Tree-AllReduce and Other AllReduce… · p. 23
  • An AllReduce can be represented as a directed graph G = (V,E), and any isomorphic graph G′ = (V,E′) can perform the operation equally well through a node permutation in Sym(V).V is the cluster's node set, E denotes data dependencies, and the homomorphism between G and G′ is described by the symmetric group on V.Source: A Tree-AllReduce and Other AllReduce… · p. 23

B Commercially Available Patch Panels and Optical Circuit Switches · C Handling Sharding and Dynamic Job Arrivals in Shared Clusters

Patch panels provide high port counts at lower cost and loss but reconfigure slowly, while optical circuit switches reconfigure faster at higher cost and loss. TOPOOPT supports shared jobs by isolating their resources and preparing the next topology in parallel with current training.

B Commercially Available Patch Panels and Optical Circuit Switches

  • 1008 duplex ports, <0.5 dB insertion loss, and ≈$100K ($100/port) are offered by Telescent reconfigurable patch panels, whose reconfiguration takes several minutes.A robotic arm connects transmit-side fibers to receive-side fibers; Telescent NTM panels have NEBS Level 3 certification and over 1 billion port hours in operation.Source: B Commercially Available Patch Panels… · p. 23
  • 384 duplex ports, ≈10 ms reconfiguration, $200K ($520/port), and 1.5–2.7 dB loss characterize the largest market OCS; versus patch panels, OCSs cost five times more, have three times fewer ports, and reconfigure four orders faster.Source: B Commercially Available Patch Panels… · p. 24

C Handling Sharding and Dynamic Job Arrivals in Shared Clusters

  • Sharding configures disjoint server-and-link groups for each job, giving each dedicated resources and benefiting performance, especially tail latency.Figure 26 illustrates a two-job cluster split into green and red groups.Source: C Handling Sharding and Dynamic Job… · p. 24, Figure 26 · p. 24
  • Because patch-panel reconfiguration can delay job starts by several minutes, TOPOOPT pre-provisions the next topology while current jobs run.Source: C Handling Sharding and Dynamic Job… · p. 24
  • $25 mechanical 1×2 switches with 0.73 dB measured loss select Active or Look-ahead ports, avoiding the 3 dB loss of optical splitters.With n servers and d interfaces per server, the design uses 2d patch panels; each interface has Active and Look-ahead parts. Figure 25 depicts the ports.Source: C Handling Sharding and Dynamic Job… · p. 24, Figure 25 · p. 24
  • Because topology and parallelization are calculated offline from the known job-arrival sequence and server needs, TOPOOPT prepares the next job’s topology and flips to it immediately once its servers are ready.Source: C Handling Sharding and Dynamic Job… · p. 24

D Model Configurations and Transfer Sizes … F Modifications to SiP-ML

These sections specify the model configurations and transfer patterns used in evaluation, then detail algorithms for generating AllReduce rings, routing traffic, optimizing reconfigurable topologies, and adapting SiP-ML. The methods use workload demands and node connectivity to guide topology and routing choices.

D Model Configurations and Transfer Sizes

  • Evaluated workloads include VGG, ResNet50, BERT, DLRM, CANDLE, and NCF; simulations use common Meta configurations, while prototype parameters and batches are reduced to fit 12 nodes.List 1 provides the model parameters and batch sizes used across simulations and the testbed.Source: D Model Configurations and Transfer… · pp. 24–25
  • For DLRM’s 20 GB embedding example, data parallelism requires 37.5 GB/server AllReduce versus 32 MB/server model-parallel traffic at batch size 8192.At 100 Gbps, the AllReduce transfer would take 3 seconds; pipeline parallelism can increase model-parallel traffic, while pure model parallelism yields sparse communication among accelerators with inter-layer dependencies.Source: D Model Configurations and Transfer… · p. 24

E.1 TOPOLOGYFINDER

  • For n-node ring-AllReduce, each integer p < n coprime with n defines a unique ring-generation rule, and changing the starting server leaves the ring equivalent.The rule advances server indices by p modulo n; the standard ring is the p = 1 case.Source: §E.1 · p. 25

E.2 Bounding maximum hop count with TotientPerms

  • With integer x, the geometric-sequence analysis gives maximum κ(n−1) = dx; when d√n < 2, choosing x = 2 yields an O(log2 n) bound.The analysis frames the hop-count bound using a geometric sequence of candidate values.Source: §E.2 · pp. 25–26

E.3 Coin Change Routing

  • CoinChangeMod routes between servers by treating selected AllReduce permutation values as coin values and the modulo-n server-index difference as the target.It modifies the classical coin-change problem to account for server IDs wrapping around the ring.Source: §E.3 · p. 26

E.4 OCS-reconfig Heuristic

  • The OCS-reconfig utility balances bandwidth for high-demand pairs against indirect-routing latency by favoring parallel links with diminishing returns.The heuristic recomputes topology every 50 ms from unsatisfied demand, pauses flows for 10 ms to represent reconfiguration, and uses two-edge replacement to preserve connectivity for host-based forwarding.Source: §E.4 · p. 26

F Modifications to SiP-ML

  • For TOPOOPT’s non-ring physical topology, the authors replace SiP-ML’s runtime-prohibitive ILP with an Algorithm 5 heuristic using Discount = 1.The adaptation targets a utility function similar to TOPOPT’s without its Discount term; the passage notes that SiP-ML’s original ring-based optimization performed poorly in this setup.Source: F Modifications to SiP-ML · pp. 26–27

G Cost of Network Components

This section describes the component prices and accounting assumptions used to estimate network costs for TopoOpt and comparison networks.

G Cost of Network Components

  • Table 2 summarizes component costs, sourced from retailer prices for NICs, transceivers, and electrical ports and supplier quotes for patch panels and optical switches.The estimates use Mellanox ConnectX NICs, discounted Telescent and Polatis pricing, and Sirius transceiver costs; 200 Gbps uses cheaper or available 100 Gbps components, while 4200 G transceivers and ports are estimated at 2× 100G cost.Source: Table 2 · p. 27, G Cost of Network Components · p. 27
  • Fat-tree and Ideal Switch costs use full-bisection Fat-tree switch counts, while a 128-node TopoOpt system of degree d uses 128×d NICs and transceivers and 128×2×d patch-panel ports.TopoOpt costing includes 2d patch panels and 1×2 switches per link for look-ahead; OCS-reconfig assumes d OCSs connected to all servers in a flat topology.Source: G Cost of Network Components · p. 27
  • Fiber-optic cables are estimated at 30 cents per meter with lengths uniformly distributed from 0 to 1000 meters, and optical-component costs remain constant as link bandwidth increases.Source: G Cost of Network Components · p. 27

H Impact of Server Degree on TOPOOPT’s Performance

Increasing server degree benefits network-heavy models, while BERT’s iteration time changes marginally. At degree eight, the trends across network architectures remain consistent despite increased per-server bandwidth.

H Impact of Server Degree on TOPOPT’s Performance

I Enabling Host-based Forwarding in RDMA

TopoOpt enables host-based RoCEv2 forwarding by splitting NIC interfaces and configuring Linux forwarding rules to create logical RDMA connections between servers. The approach has negligible overhead when forwarded traffic is small, though it is slower than native point-to-point RDMA.

I Enabling Host-based Forwarding in RDMA

  • RoCEv2 packets carry routable IP addresses, but the NIC silently drops packets whose destination IP does not match its own, preventing ordinary host-based forwarding.Packet processing and memory access are offloaded to the NIC, leaving the host without access to individual packets.Source: I Enabling Host-based Forwarding in RDMA · p. 28
  • NPAR splits each 25 Gbps physical interface into an IP-addressed RDMA interface and a separate interface that directs forwarded traffic to the host networking stack.The second interface has a distinct MAC address and RDMA disabled; traffic sent to that address bypasses the NIC’s RDMA engine.Source: I Enabling Host-based Forwarding in RDMA · p. 28
  • Linux iproute, arp, and tc flower rules route multihop traffic by final destination IP, forwarding packets through intermediate hosts and rewriting their next-hop MAC addresses.The source and destination servers select outgoing ports and next-hop MACs, while intermediate servers use tc flower rules to apply the algorithm’s route.Source: I Enabling Host-based Forwarding in RDMA · pp. 28–29
  • The forwarding rules create logical RDMA connections between every server pair, and topology-aware NCCL uses them to meet its all-to-all connectivity requirement.NCCL is modified to account for server pairs that connect only through specific ports.Source: I Enabling Host-based Forwarding in RDMA · p. 29
  • Host-based forwarding is slower than native point-to-point RDMA, but experiments show negligible overhead when forwarded traffic is small.The NICs support TCP forwarding offload; firmware and driver changes or future NIC versions could add RDMA forwarding offload and reduce overhead further.Source: I Enabling Host-based Forwarding in RDMA · p. 29

Related papers

Loading 2202.00433v3…