PaperlayerView exact revision on arXiv

Source-linked AI summary

TopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed Training Jobs

Weiyang Wang, Moein Khazraee, Zhizhen Zhong, Manya Ghobadi, Zhihao Jia, Dheevatsa Mudigere, Ying Zhang, Anthony Kewitsch

arXiv:2202.00433v3cs.NIcs.DC

TL;DR

Distributed DNN training increasingly strains conventional networks, while traffic depends on parallelization strategy and can be predictable across iterations. TOPOOPT co-optimizes computation, communication, and topology using alternating optimization and TotientPerms, implemented in a direct-connect optical fabric. It reduces training iteration time by up to 3.4× versus a similar-cost Fat-tree, while remaining subject to traffic-stability and link-failure limitations.

  • Problem

    Distributed DNN training faces increasing network overhead, while prior approaches do not optimize physical topology alongside computation and communication.

  • Method

    TOPOOPT uses a shardable optical direct-connect fabric, alternating optimization, and TotientPerms to jointly select parallelization strategy, topology, and routing.

  • Results

    Up to 3.4× faster training iteration time is achieved than with a similar-cost Fat-tree across six representative DNN models.

  • Takeaways & Limitations

    Predictable DNN traffic enables demand-aware network topology optimization for distributed training jobs.

  • Takeaways & Limitations

    TOPOOPT assumes traffic patterns remain unchanged between iterations, and a single link failure can make an AllReduce ring inefficient.

Abstract

from arXiv · show

We propose TopoOpt, a novel direct-connect fabric for deep neural network (DNN) training workloads. TopoOpt co-optimizes the distributed training process across three dimensions: computation, communication, and network topology. We demonstrate the mutability of AllReduce traffic, and leverage this property to construct efficient network topologies for DNN training jobs. TopoOpt then uses an alternating optimization technique and a group theory-inspired algorithm called TotientPerms to find the best network topology and routing plan, together with a parallelization strategy. We build a fully functional 12-node direct-connect prototype with remote direct memory access (RDMA) forwarding at 100 Gbps. Large-scale simulations on real distributed training models show that, compared to similar-cost Fat-Tree interconnects, TopoOpt reduces DNN training time by up to 3.4x.

1 Introduction

Distributed DNN training is increasingly network-bound, while existing approaches optimize computation and communication without optimizing physical topology. TOPOOPT addresses this gap by jointly optimizing parallelization strategy, communication, and network topology.

  • Growing DNN model and dataset requirements create urgent demand for efficient distributed training systems.
  • Fat-tree interconnects are becoming a bottleneck for distributed DNN training workloads.They were designed for unpredictable traffic and uniform server-to-server bandwidth and latency.
  • Prior methods co-optimize computation and communication but omit physical network topology as an optimization dimension.
  • Communication overhead increases dramatically with worker count, and traffic patterns depend on parallelization strategy.
  • TOPOOPT jointly optimizes topology and parallelization strategy using reconfigurable optical partitions for each training job.
  • Alternating optimization searches parallelization and topology iteratively, while TotientPerms exploits mutable AllReduce traffic to support efficient topology construction.The topology must handle large AllReduce transfers efficiently and keep Model Parallel transfers short-hop.
  • 3.4× lower training iteration time is achieved than a similar-cost Fat-tree in a 12-server prototype evaluation.The prototype uses 100 Gbps networking and RDMA forwarding.

2 Motivation

Distributed DNN jobs at Meta are large, long-running, and increasingly communication-intensive, with traffic shaped by parallelization choices. These predictable patterns motivate demand-aware, topology-optimized training networks.

  • Long-lasting jobs make demand-aware networks practical because their traffic distributions can be pre-computed before execution.
  • Hybrid DLRM parallelism reduces the maximum transfer size from 44 GB to 4 GB by partitioning embedding tables while replicating the remaining model.
  • DLRM traffic combines large AllReduce transfers with one-to-many and many-to-one Model Parallel transfers.In the example, AllReduce transfers are 4 GB and Model Parallel transfers are 32 MB.
  • Production DNN jobs commonly use 32 to 700 workers, and most run for more than 10 hours.The top 10% of jobs run for more than 96 hours.
  • Network communication accounts for up to 60% of a DNN training iteration as GPU count increases from 8 to 128.The measurements use RDMA and define overhead as communication time during training.
  • Ring-AllReduce produces recurring diagonal traffic patterns, while Model Parallel traffic depends on parallelization strategy and device placement.
  • Production training traffic remains identical across iterations because jobs retain the same parallelization strategy and synchronization method.This creates periodic and predictable per-iteration heatmaps.

3 TOPOOPT System Design

TOPOOPT is a shardable direct-connect fabric that uses optical switching to create job-specific server partitions and reconfigure their interconnects. It supports direct connections, host-based forwarding, and multiple optical-switch technologies for distributed DNN training.

  • TOPOOPT interconnect: TOPOOPT jointly optimizes DNN parallelization strategy and network topology within dedicated job partitions.The system uses optical devices to create demand-aware interconnects for individual training jobs.
  • TOPOOPT interconnect: Each server has d interfaces connected to a core layer of d optical switches, and shard size follows the servers requested by the job.The prototype uses one 100 Gbps NIC with 4×25 Gbps interfaces, giving degree four.
  • TOPOOPT interconnect: Optical switches shard the cluster into dedicated partitions, after which TOPOOPT finds the strategy and topology offline and reconfigures the switches before training.The target topology is realized for the job after offline optimization.
  • Forwarding and scaling: Host-based forwarding lets servers forward traffic when the server degree is smaller than the number of communicating neighbors.Hosts act as switches and forward incoming traffic toward its destination.
  • Optical switching technologies: TOPOOPT can use optical patch panels or optical circuit switches, with the choice depending on cluster scale, job iteration time, and job-arrival frequency.OCSs may support within-training reconfiguration, whereas patch panels suit topologies that remain intact throughout a job.

4 Co-optimizing Parallelization Strategy and Network Topology

TOPOOPT makes joint optimization tractable by alternating between parallelization and topology searches, while exploiting mutable AllReduce traffic to build efficient combined topologies and routes.

  • 4.1 Alternating Optimization: Jointly optimizing computation, communication, and topology is difficult because the search space expands beyond the already NP-complete parallelization problem.A cross-layer formulation can become impractical even at modest scales, such as six nodes.
  • 4.1 Alternating Optimization: TOPOOPT alternates between searching parallelization for a fixed topology and finding a topology for the resulting traffic until convergence.The framework divides the search into Comp.×Comm. and Comm.×Topo. planes and uses FlexFlow’s MCMC search for parallelization.
  • 4.2 TOPOLOGYFINDER: TOPOLOGYFINDER takes dedicated servers, server degree, AllReduce transfers, and model-parallel transfers, then returns a topology and routing rules.The returned topology and routes feed the next round of the alternating optimization.
  • 4.2 TOPOLOGYFINDER: TOPOLOGYFINDER allocates server degree between AllReduce and model-parallel sub-topologies according to their traffic shares, while reserving at least one degree for AllReduce.AllReduce groups receive degree proportionally to their traffic, and model-parallel connectivity is built with maximum-weight matching.
  • 4.3 Traffic Mutability and AllReduce Topology: AllReduce traffic is mutable: server-order permutations preserve correctness and latency while potentially reducing model-parallel hop counts.TOPOPT overlaps multiple ring-AllReduce permutations to balance AllReduce transfers and provide short model-parallel paths.
  • 4.3 Traffic Mutability and AllReduce Topology: TOPOPT bounds the AllReduce sub-topology diameter by O(dA · n1/dA) under specified assumptions.For DLRM, combining three ring-AllReduce permutations produces a more balanced traffic matrix while retaining short model-parallel hop counts.

5 Large Scale Simulations

Large-scale simulations compare TOPOOPT with alternative interconnects across dedicated and shared DNN-training clusters. TOPOOPT often improves iteration time, but all-to-all traffic, host-based forwarding, routing imbalance, and reconfiguration latency constrain its benefits.

  • Methodology & Setup: TOPOOPT simulations use FlexNet and evaluate fixed-shard, dynamically reconfigurable, Ideal Switch, Fat-tree, and other interconnects.FlexNet augments FlexFlow with network awareness; the evaluation includes dedicated and shared-cluster settings.
  • Methodology & Setup: 1.33× higher average cost makes OCS-based TOPOOPT more expensive than patch-panel implementations, while TOPOOPT overlaps Fat-tree cost.The cost comparison intentionally matches TOPOOPT and Fat-tree costs.
  • Performance Comparison on Dedicated Clusters: 2.8× faster CANDLE iteration time, 2.8× faster VGG iteration time, and 3× faster BERT iteration time are reported versus Fat-tree.CANDLE is evaluated on a 128-server dedicated cluster; VGG and BERT show similar trends.
  • Performance Comparison on Dedicated Clusters: 2.8× and 2.1× faster iteration times are reported for DLRM and NCF versus Fat-tree, while Ideal Switch is further 1.3× and 1.7× faster than TOPOOPT.TOPOOPT diverges more from Ideal Switch for these models because host-based forwarding handles many-to-many model-parallel transfers.
  • Impact of All-to-all Traffic: 2.7× slower Fat-tree performance occurs at batch size 128, when TOPOOPT matches Ideal Switch for DLRM.The top axis reports the all-to-all-to-AllReduce traffic ratio for each batch size.
  • Impact of All-to-all Traffic: 1.1× higher TOPOOPT iteration time than Fat-tree occurs at batch size 2048 with all-to-all traffic equal to 80% of AllReduce traffic.Increasing server degree mitigates this degradation, while the evaluated workload contains more all-to-all traffic than a cited industry model.
  • Impact of Host-based Forwarding: 3.03 bandwidth tax at batch size 2048 and d = 4 can make TOPOOPT slower than Fat-tree, with tolerable tax depending on compute and overlap.The tolerable bandwidth tax varies across DNN models because compute time and compute-communication overlap differ.
  • Impact of Host-based Forwarding: 5.7 average path length at d = 4 implies at least 5.7× forwarding overhead for all-to-all traffic relative to Ideal Switch.Because TOPOOPT has higher total network bandwidth than Fat-tree, the forwarding overhead becomes at least 1.4× for the Fat-tree comparison.

6 Prototype

The prototype implements TOPOOPT on a 12-server RDMA testbed and evaluates throughput, time-to-accuracy, and all-to-all traffic effects.

  • Testbed setup: The prototype uses 12 servers, each with one A100 GPU and 100 Gbps networking, supporting degree d = 4 and B = 25 Gbps.The HP NICs provide four 25 Gbps interfaces through breakout fibers.
  • Distributed training framework: FlexFlow trains ResNet50, BERT, VGG16, and CANDLE, while DLRM uses Facebook’s implementation.The prototype uses smaller model and batch sizes than the simulation setup.
  • Modifications to NCCL: TOPOOPT modifies NCCL to respect topology-aware routing and integrate TotientPerms AllReduce permutations.The changes also enable load-balancing parameter synchronization across multiple ring-AllReduce permutations.
  • RDMA forwarding: RDMA forwarding is enabled by NIC firmware and driver updates that route forwarded RoCEv2 packets without kernel processing.The implementation uses network partitioning to separate host-based forwarding traffic from direct traffic.
  • Training performance: TOPOOPT reaches 90% accuracy on VGG19/ImageNet 2.0× faster than the Switch 25Gbps baseline and performs similarly to the Switch 100Gbps baseline.Its training throughput is similar to the Switch 100 Gbps baseline across all evaluated models.
  • Impact of all-to-all traffic: The testbed measures average iteration time across 320 DLRM iterations while varying all-to-all model-parallel traffic through batch size and enlarged embedding dimensions.Embedding dimensions are increased by 128× to create worst-case traffic.

7 Discussion

The discussion defines TOPOOPT’s target workload and operational scope, describes extensions for elasticity and failures, and identifies limitations and broader applicability.

  • Target workload: TOPOOPT targets large DNN jobs using hybrid data and model parallelism, with fixed server assignments and no GPU sharing between jobs.The target workload assumes each job retains its assigned servers throughout its lifetime.
  • Storage and control plane traffic: TOPOOPT partitions GPU training traffic, while CPU NICs use a separate fabric for storage and other traffic.Only GPU NICs count toward server degree.
  • Supporting dynamic scheduling and elasticity: Elasticity requires fast-reconfigurable optical switches, and jointly maintaining optimal topology and parallelization after server changes remains future work.Adding dynamic server membership expands the optimization space with another dimension.
  • Handling failures: A single link failure does not disconnect TOPOOPT’s graph; temporary recovery can use an MP-dedicated link, while permanent failures trigger port swapping.These mechanisms recover failed connections without relying on a single ring topology.
  • Supporting multi-tenancy: TOPOOPT can support multi-tenancy by using NVIDIA MIG to represent one physical server as multiple logical servers.This maps physical resources into separate topology participants.
  • TotientPerms in Fat-trees: TotientPerms may also benefit Fat-tree networks because distributing AllReduce traffic across permutations can reduce congestion.The technique is not limited to reconfigurable optical interconnects.
  • TOPOOPT’s limitations: TOPOOPT assumes traffic patterns remain unchanged between iterations, which may not hold for GNN or MoE models.A single link failure within an AllReduce ring also makes that ring inefficient until optical reconfiguration occurs.

8 Related Work

Prior systems optimize DNN parallelization, scheduling, or datacenter interconnects separately, whereas TOPOOPT jointly optimizes parallelization and physical topology for DNN traffic.

  • Optimizing DNN training: Existing DNN optimization frameworks distribute data or models while considering bandwidth but generally do not optimize the physical-layer topology.TOPOOPT addresses the topology dimension alongside training strategy.
  • DNN parallelization strategies: Automated parallelization systems search broad strategy spaces but are designed for conventional Fat-tree interconnects.Examples include FlexFlow, ColocRL, and MERLIN.
  • DNN training infrastructures and schedulers: Existing training infrastructures use nonreconfigurable topologies such as Fat-tree and Torus, while TOPOOPT uses commodity reconfigurable interconnects.Cluster scheduling approaches are complementary and can provide techniques for TOPOOPT clusters.
  • Optical Interconnects: Optically reconfigurable and traffic-oblivious datacenter networks can produce sub-optimal topologies for repetitive DNN training traffic.TOPOOPT instead adapts topology to distributed DNN traffic demands.

9 Conclusion

TOPOOPT jointly explores computation, communication, and topology choices, using AllReduce traffic mutability to construct effective communication structures. Its design supports efficient AllReduce while accommodating model-parallel transfers.

  • Conclusion: TOPOOPT explores the Computation × Communication × Topology strategy space with alternating optimization and achieves up to 3.4× faster training iteration time than Fat-tree.The system jointly optimizes DNN parallelization strategy and network topology using optical devices.
  • A Tree-AllReduce and Other AllReduce Permutations: Tree-AllReduce reduces data to a root using recursive halving, then broadcasts results using recursive doubling.The operation is represented as a logical tree among servers.
  • A Tree-AllReduce and Other AllReduce Permutations: Double binary trees balance communication by constructing two trees with flipped leaf and in-tree labeling.This gives each non-root node the same AllReduce communication requirements.
  • A Tree-AllReduce and Other AllReduce Permutations: Permuting node labels across a pair of double binary trees creates alternative trees that perform AllReduce at the same speed.The resulting permutations can produce different traffic demand matrices for workloads such as DLRM and CANDLE.
  • A Tree-AllReduce and Other AllReduce Permutations: AllReduce can be represented as a directed dependency graph, and any isomorphic relabeling can perform the operation equally well.The relabelings are described by the symmetric group on the cluster nodes.

B Commercially Available Patch Panels and Optical Circuit Switches

The section compares commercially available optical patch panels and optical circuit switches, then describes how TopoOpt uses reconfiguration and look-ahead provisioning to support isolated, multi-job clusters.

  • Patch panels support on-demand cable management and software-controlled reconfiguration, with commercial systems offering 1008 duplex ports at less than 0.5 dB insertion loss and approximately $100K cost.
  • Optical circuit switches redirect light with MEMS mirrors, but the largest marketed device has 384 duplex ports, approximately 10 ms reconfiguration latency, and approximately $200K cost.
  • OCSs have five-times-higher port cost, higher insertion loss, and three-times-lower port count than patch panels, while offering faster reconfiguration.
  • TopoOpt isolates concurrent jobs by configuring disjoint server and link subsets, giving each job dedicated resources.
  • Look-ahead provisioning preconfigures the next topology while current jobs run, using inexpensive 1×2 mechanical optical switches to select active or standby patch-panel ports.The prototype switches cost $25 and measured 0.73 dB optical loss.

D Model Configurations and Transfer Sizes

The section specifies DNN model configurations and motivates co-optimizing data and model parallelism with topology and routing, using permutation and routing algorithms for AllReduce traffic.

  • The simulations and testbed use VGG, ResNet50, BERT, DLRM, and CANDLE with model-specific batch sizes, layer dimensions, sequence lengths, and attention configurations.The listed configurations distinguish simulation, testbed, and section-specific settings.
  • 37.5 GB of AllReduce data must be sent and received per server when a 20 GB DLRM embedding table uses data parallelism.
  • TOPOPT distributes network degree between model-parallel and AllReduce sub-topologies to accommodate workloads whose communication balance varies with bandwidth.At very high bandwidth, model parallelism can reduce system overhead when model-parallel traffic becomes significant.
  • Ring-AllReduce permutations are generated by mapping each server S_i to S_(i+1) mod n, with rotations of the same ring treated as equivalent.
  • Every p<n coprime with n defines a unique ring-AllReduce permutation, and the resulting graph construction guarantees distinct rings.
  • The geometric-sequence construction targets an approximately O(d d√n) bound on maximum cluster diameter, with an O(log2 n) bound when d√n<2.
  • CoinChangeMod routes between nonadjacent servers by treating selected permutation values as modular coin denominations and minimizing the number of hops.
  • The OCS-reconfig heuristic balances bandwidth and indirect-routing latency by maximizing parallel links for high unsatisfied traffic demands with diminishing returns.Its utility function weights traffic demand by a discount based on the number of parallel links.

G Cost of Network Components

The section defines component-cost assumptions for comparing Fat-tree, Ideal Switch, and TopoOpt networks, including port, transceiver, fiber, NIC, and optical-component costs.

  • Component costs cover NICs, transceivers, fibers, electrical switches, patch panels, and optical switches, using retailer prices or supplier quotations.
  • For 200 Gbps links, the evaluation uses multiple 100 Gbps ports and fibers when those components are cheaper or high-end prices are unavailable.
  • The cost comparison includes a 4,200 Gbps estimate equal to twice the cost of 100 Gbps transceivers and switch ports.
  • A standard k=8 full-bisection Fat-tree has 80 64-port switches, totaling 640 switch ports, plus one NIC and transceiver per host-side connection.
  • A 128-node TopoOpt system with degree d uses 128×d NICs and transceivers and 128×2×d patch-panel ports because of look-ahead provisioning.

H Impact of Server Degree on TOPOOPT’s Performance

Increasing server degree raises TopoOpt bandwidth and can reduce topology diameter, benefiting network-heavy models more than compute-bound BERT at higher bandwidth.

  • With degree d=8, increased per-server bandwidth preserves the relative behavior of the evaluated network architectures.
  • DLRM and CANDLE benefit most from increasing server degree because both are network-heavy workloads.
  • CANDLE improves almost linearly with degree because its strategy is closer to data parallelism and requires more bandwidth.
  • Higher degree lets TOPOLOGYFINDER find lower-diameter topologies, improving performance through greater bandwidth and fewer hops for model-parallel transfers.
  • BERT shows marginal iteration-time impact from increased degree and bandwidth because it is mostly compute bound at higher bandwidth.

I Enabling Host-based Forwarding in RDMA

TopoOpt enables host-based forwarding for multihop RDMA by steering forwarded traffic through the host networking stack while preserving RDMA interfaces for endpoint traffic. Routing rules and topology-aware NCCL provide logical RDMA connectivity across all servers, with negligible overhead when forwarded traffic is small.

  • Interface-based forwarding: NPAR splits each 25 Gbps physical interface into two logical interfaces, directing forwarded traffic through the host networking stack instead of the NIC’s RDMA engine.if1 remains a normal RDMA interface with an IP address, while if2 has no IP address or RDMA and uses a distinct MAC address.
  • Routing configuration: Linux iproute, ARP, and tc flower rules select outgoing ports and next-hop MAC addresses for multihop packet forwarding.Direct connections use outgoing-interface selection, while multihop paths deliver packets to the kernel for forwarding at intermediate servers.
  • Packet walk-through: A packet from server A to server D traverses servers B and C as Ethernet traffic before being converted back into an RDMA packet at server D.The reverse path applies the same forwarding process in the opposite direction, enabling bidirectional communication.
  • System integration: The forwarding rules create logical RDMA connections between every server pair, enabling all-to-all communication libraries such as NCCL.NCCL is modified to account for topology-specific port connectivity.
  • Performance boundary: Forwarded traffic incurs a performance penalty, but experiments indicate negligible overhead when the forwarded-traffic volume is small.Future RDMA forwarding offload in firmware and drivers could further reduce this overhead.
Loading 2202.00433v3…