Source-linked AI summary
NegotiaToR: Towards A Simple Yet Effective On-demand Reconfigurable Datacenter Network
Cong Liang, Xiangli Song, Jing Cheng, Mowei Wang, Yashe Liu, Zhenhua Liu, Shizhen Zhao, Yong Cui
TL;DR
Dynamic datacenter traffic demands high goodput and low latency, while existing packet-switched networks face capacity constraints and traffic-oblivious optical designs can incur relay costs. NegotiaToR uses distributed, in-band on-demand scheduling with binary demand information and an incast-oriented delay bypass. Simulations report better mice-flow FCT and goodput than the state-of-the-art traffic-oblivious design on two representative flat topologies.
Problem
Method
NegotiaToR distributes on-demand scheduling among ToRs using binary demand information and an in-band control plane, with a scheme to bypass scheduling delays during incasts.
Source: §1 · p. 1, §3 · p. 3, §3.1 · p. 3Results
One to two orders of magnitude better mice-flow FCT than the state-of-the-art traffic-oblivious design, alongside better FCT and goodput across two representative flat topologies.
Source: §1 · p. 2Takeaways & Limitations
NegotiaToR is compatible with prevalent flat topologies and uses a minimalist design intended to enhance practicality.
Source: §6 · p. 12For a previously empty source-destination pair, scheduling delay of at least two epochs is inevitable, and additional waiting occurs if the request is not granted.
Source: §3.3.2 · p. 6
Abstract
from arXiv · showhide
Recent advances in fast optical switching technology show promise in meeting the high goodput and low latency requirements of datacenter networks (DCN). We present NegotiaToR, a simple network architecture for optical reconfigurable DCNs that utilizes on-demand scheduling to handle dynamic traffic. In NegotiaToR, racks exchange scheduling messages through an in-band control plane and distributedly calculate non-conflicting paths from binary traffic demand information. Optimized for incasts, it also provides opportunities to bypass scheduling delays. NegotiaToR is compatible with prevalent flat topologies, and is tailored towards a minimalist design for on-demand reconfigurable DCNs, enhancing practicality. Through large-scale simulations, we show that NegotiaToR achieves both small mice flow completion time (FCT) and high goodput on two representative flat topologies, especially under heavy loads. Particularly, the FCT of mice flows is one to two orders of magnitude better than the state-of-the-art traffic-oblivious reconfigurable DCN design.
1 INTRODUCTION
NegotiaToR addresses the challenge of practical on-demand scheduling for dynamic datacenter traffic with a minimalist, distributed design. Simulations report better mice-flow FCT and goodput than a traffic-oblivious design.
1 INTRODUCTION
- NegotiaToR uses existing AWGRs and fast-tunable lasers and is compatible with prevalent flat topologies.It dynamically adapts optical links among ToR switches and sends data directly to destinations through one-hop paths.Source: §1 · p. 1
- Dynamic and unpredictable traffic makes on-demand scheduling attractive, but prior optical solutions raise practicality concerns from scheduling complexity.Existing packet-switched DCNs also struggle to meet goodput and latency requirements because of inadequate switching-chip capacity.Source: §1 · p. 1
- NegotiaToR distributes scheduling across ToRs using binary demand information, avoiding the need for detailed data-size measurements.Each ToR schedules its ingoing and outgoing traffic.Source: §1 · p. 1
- An in-band control plane lets ToRs exchange scheduling messages over periodic all-to-all connectivity, avoiding a separately deployed control network.Scheduling runs in a pipelined manner, and senders access results locally.Source: §1 · p. 2
- To mitigate scheduling delays for latency-sensitive mice flows, NegotiaToR piggybacks small volumes of data during periodic all-to-all connectivity, including during incasts.Mice flows often arrive simultaneously as incasts in partition/aggregate applications.Source: §1 · p. 2
- Mice-flow FCT is one to two orders of magnitude better than the state-of-the-art traffic-oblivious design, with better FCT and goodput under comparable complexity.The evaluation uses two representative flat topologies.Source: §1 · p. 2
2 BACKGROUND & MOTIVATION
Fast optical switching can support direct rack-to-rack connections, but dynamic, unpredictable traffic makes inter-ToR scheduling difficult. Flat AWGR topologies offer high- and low-port-count options, while traffic-oblivious relay can reduce goodput and worsen mice-flow FCT.
2 BACKGROUND & MOTIVATION
- AWGRs forward data according to the source laser's wavelength, and fast optical switching supports reconfiguration for dynamic ToR traffic.Flat one-layer AWGR topologies are common because of insertion loss and wavelength limitations in cascaded AWGRs.Source: §2 · p. 2
- A single high-port-count AWGR can connect all ToRs in a parallel topology; thin-clos is an alternative when low-port-count AWGRs cannot do so alone.In thin-clos, each ToR port reaches a subset of ToRs, while all ports together reach the whole network.Source: §2 · p. 2, Figure 1 · p. 2
- Inter-ToR traffic is highly dynamic and unpredictable, including bursty incasts, challenging centralized scheduling scalability and motivating adaptive approaches.Traffic-oblivious designs instead reconfigure in a round-robin manner regardless of actual traffic patterns.Source: §2 · pp. 2–3
- Traffic relay in traffic-oblivious designs doubles traffic volume, can reduce worst-case goodput to 50%, and can damage mice-flow FCT, especially under heavier loads.Relayed elephant flows may block mice flows at intermediate nodes.Source: §2 · p. 3
3 NEGOTIATOR DESIGN
This section presents NegotiaToR’s minimalist, distributed on-demand optical DCN design, in which ToRs use in-band binary demand-based scheduling to send traffic over one-hop optical links. It justifies the scheduling choices and describes mechanisms for link failures, scheduling implementation, reconfiguration overhead, and traffic below ToRs.
3.1 Design overview
- The architecture supports flat topologies such as parallel and thin-clos, using buffered ToRs, bufferless optical links, and per-destination queues for one-hop traffic.ToR uplinks use fast-tunable lasers attached to AWGR-based optical switches.Source: §3 · p. 3, §3.1 · p. 3
3.3.1 In-band pipelined scheduling.
- Each epoch has an all-to-all predefined control phase and a scheduled data phase; pipelining REQUEST, GRANT, and ACCEPT across epochs needs one all-to-all round per epoch and about two epochs minimum per schedule.A single-epoch process would require three round-robin rounds and incur long scheduling delays, so NegotiaToR pipelines the steps.Source: §3.1 · p. 3, §3.3.1 · p. 6
3.2.1 NegotiaToR Matching algorithm.
- NegotiaToR Matching uses three non-iterative REQUEST, GRANT, and ACCEPT steps on binary per-destination demands, locally resolving source- and destination-port conflicts without traffic forecasting.Round-robin GRANT and ACCEPT rings promote fairness and avoid starvation in port allocation.Source: §3.2.1 · p. 4
3.2.2 Matching efficiency.
- Under intense conflicts, NegotiaToR Matching has 63% matching efficiency; thin-clos typically has higher efficiency because its limited connectivity reduces competing grants.The 63% figure is the limiting acceptance chance in the simplified high-load model for the parallel topology.Source: §3.2.2 · p. 5
3.3.2 One-hop data transmission.
- Scheduled one-hop transmission serves queued demand, but newly active source-destination pairs face at least two epochs of scheduling delay, with extra waiting if their request is not granted.This delay is acceptable for elephant flows but can harm mice-flow FCT, especially during incasts.Source: §3.3.2 · p. 6, §3.4 · p. 6
3.4.1 Data piggybacking in the predefined phase.
- Piggybacking one small data packet with scheduling messages guarantees each ToR pair at least one packet per epoch, letting mice flows bypass scheduling delay while keeping the predefined phase short.Most epoch connections remain available in the scheduled phase for demand-based transmission and goodput.Source: §3.4.1 · p. 6
3.4.2 Priority queue for mice flow prioritization.
- ToR mice-flow priority queues using PIAS let mice data transmit ahead of elephants in both phases, reducing mice FCT, most of which falls below scheduling delay.PIAS uses multilevel feedback queues without requiring prior knowledge of flow sizes.Source: §3.4.2 · p. 6
3.5 The rationality of NegotiaToR’s design choices
- Iteration yielded minimal or negative goodput gains and worse FCT, while 2× link-rate speedup achieved high goodput and superior FCT; selective relay gave minor or no goodput gain.The selective-relay evaluation was on thin-clos; it barely helped at heavy loads, while NegotiaToR already had good goodput at light loads.Source: §3.5 · p. 7
- Simulations found only modest gains from priority-enriched requests and negligible differences between stateful and stateless scheduling, supporting NegotiaToR’s minimalist binary-request design.Priority approaches favored pairs with more pending data or longer head-of-line waits; stateless scheduling avoids state-maintenance complexity, with duplicate requests having minor impact in the evaluated load conditions.Source: §3.5 · p. 7
3.6.1 Fault tolerance.
- Dummy scheduling messages and port-level feedback help ToRs detect ingress or egress link failures, exclude faulty links from scheduling, and restore them after repair.For parallel topologies, rotating the predefined-phase round-robin rule lets ToR pairs exchange scheduling messages over alternate links; thin-clos can relay through an intermediate ToR, while upper-layer protocols such as TCP retransmit lost packets.Source: §3.6.1 · p. 8
3.6.2 Scalable scheduling logic implementation.
- NegotiaToR Matching can use programmable priority encoders to implement round-robin GRANT and ACCEPT rings, distributing scheduling logic across ToRs for scalable implementation.Related switch matching algorithms can complete one iteration in several clock cycles even for switches with hundreds of ports; NegotiaToR has a longer scheduling-time budget than an individual switch.Source: §3.6.2 · p. 8
3.6.3 End-to-end reconfiguration delay.
- Current technologies can support a 10-nanosecond end-to-end reconfiguration delay, while NegotiaToR remains effective with a 100-nanosecond guardband.Fast wavelength tuning and clock/data recovery, plus precise synchronization, can reduce guardband needs; the cited synchronization example achieves errors within picoseconds.Source: §3.6.3 · p. 8
3.6.4 Epoch length setting considerations.
- Epoch lengths must balance predefined-phase bypass opportunities against scheduled-phase duration, scheduling delay, stale results, link waste, and guardband overhead.The design targets guardbands of 10% or less of total time; the evaluation uses default epoch settings and examines guardband and epoch-length sensitivity.Source: §3.6.4 · pp. 8–9
3.6.5 Traffic management below ToRs.
- Below ToRs, backpressure or credit-based flow control can prevent drops, while receiver-side buffer monitoring can limit arrivals when queues lack space.Ordering packets from per-destination to per-port queues and consuming them in the same order at the destination can preserve packet order for a source-destination pair.Source: §3.6.5 · p. 9
4 EVALUATION
The evaluation uses large-scale packet-level simulations on two flat topologies, testing NegotiaToR’s design elements and overall performance across loads and workloads. It reports advantages in mice-flow FCT and high-load goodput over traffic-oblivious scheduling, alongside robustness across scenarios.
4.1 Evaluation setup
- Under the tested setup, simulations use 128 8-port ToRs on parallel and thin-clos topologies, Hadoop-derived workloads, and measurements of 99th-percentile mice-flow FCT and average goodput.Unless specified otherwise, data piggybacking and priority queues are enabled; mice flows are under 10 KB, and loads range from 10% to 100%.Source: §4.1 · p. 9, §4 · p. 9
4.2 Microbenchmarks
- At 100% load, over 80% of mice flows finish within 2 epochs, and enabling both data piggybacking and priority queues reduces average mice-flow FCT to 1.6 epochs.The average is below the roughly 2-epoch scheduling delay; data piggybacking alone also reduces 99th-percentile mice-flow FCT relative to no optimization.Source: §4.2 · p. 10, Table 2 · p. 10, Figure 6 · p. 10
- NegotiaToR finishes incasts earlier than the traffic-oblivious scheme at roughly the same time across incast degrees, while achieving higher goodput in heavy all-to-all loads.The parallel topology’s goodput exceeds thin-clos’s because it maintains higher link utilization; relayed traffic limits the traffic-oblivious scheme’s goodput, especially at heavy loads.Source: §4.2 · p. 10, Figure 7 · p. 10
4.3 Main results
- With priority queues enabled, NegotiaToR’s mice-flow FCT is one to two orders of magnitude better than the traffic-oblivious scheme, while its goodput is higher at high loads on both topologies.Without priority queues, significant FCT advantages remain at lighter loads; thin-clos performance is marginally lower than parallel-network performance because of limited connectivity.Source: §4.3 · pp. 10–11, Figure 9 · p. 11
- On the parallel topology, bandwidth usage falls to 98.9% with 1% link failures and 75.3% with 10% failures, then returns to its pre-failure level after recovery.A single ingress or egress link failure can reduce bandwidth disproportionately because it affects all traffic passing through that link.Source: Figure 10 · p. 11, §4.3 · p. 11
4.4 Deep dive results
- Without 2× speedup, NegotiaToR still exploits constrained bandwidth for good performance; its performance also changes only modestly when phase-length parameters are set approximately around optimal values.The parameter sensitivity experiments vary predefined-phase timeslots from 20 to 120 ns and scheduled-phase length from 10 to 500 timeslots.Source: §4.4 · p. 11, Figure 11 · p. 11, Figure 12 · p. 11
- With incasts taking 2% of ToR downlink bandwidth, NegotiaToR serves them with minor impact on background traffic; it also retains FCT and goodput advantages on web-search and Google datacenter workloads without parameter fine-tuning.The tested web-search workload has more than 80% of flows above 10 KB, while more than 80% of Google datacenter workload flows are below 1 KB.Source: §4.4 · pp. 11–12, Figure 13 · p. 12
5 RELATED WORK
Prior work addresses reconfigurable DCN scheduling through matching, routing, and traffic-oblivious designs. NegotiaToR positions itself as a minimalist on-demand approach tailored to fast switching and flat topologies.
5 RELATED WORK
- NegotiaToR adapts matching to multi-ported ToRs and different flat topologies, unlike switch-port matching algorithms; its pipelining and delay bypassing accommodate long ToR-to-ToR RTTs.Packet-switched request-grant schemes may use over-commitment, whereas reconfigurable DCNs require conflict-free scheduling.Source: §5 · p. 12
- Compared with ProjecToR’s per-port requests and source delay measurement, NegotiaToR uses binary per-ToR requests without delay measurement as a minimalist alternative.The passage describes ProjecToR’s granularity as leading to minor scalability and identifies its delay measurement as an added complexity.Source: §5 · p. 12
- Fast switching makes on-demand scheduling practical for dynamic traffic, but prior proposals raise scalability concerns, while traffic-oblivious relay can impair goodput under heavy loads and mice-flow FCT.NegotiaToR instead combines distributed matching, pipelined scheduling, and incast-oriented delay bypassing for one-hop transmission.Source: §5 · p. 12
6 CONCLUSION
The conclusion presents NegotiaToR as a minimalist on-demand architecture for reconfigurable DCNs. It uses distributed in-band scheduling over two-phase epochs, includes an incast-oriented scheduling-delay bypass, and is compatible with prevalent flat topologies.
6 CONCLUSION
- NegotiaToR runs distributed NegotiaToR Matching in-band within a two-phase epoch to reconfigure the network for dynamic real-time traffic demands.The architecture is tailored toward a minimalist design intended to enhance practicality.Source: §6 · p. 12
- Its incast-optimized scheduling-delay bypassing scheme mitigates the impact of scheduling delays.Source: §6 · p. 12
- NegotiaToR is compatible with prevalent flat topologies and is designed to exploit fast optical switching with low complexity.The authors state that they hope the design can facilitate next-generation reconfigurable DCNs.Source: §6 · p. 12
A APPENDIX
The appendix is supporting material that has not been peer-reviewed.
A APPENDIX
- The appendix is identified as supporting material that has not been peer-reviewed.Source: A APPENDIX · p. 14
A.1 Efficiency analysis validation of NegotiaToR Matching
Large-scale simulations at 100% load validate the expected efficiency of NegotiaToR Matching: thin-clos achieves a slightly higher match ratio than the parallel network topology.
A.1 Efficiency analysis validation of NegotiaToR Matching
- At 100% load, match ratio is 0.644 for thin-clos (n=16) and 0.634 for the parallel network topology (n=128), consistent with expectations.The simulations test scheduling efficiency under heavy load, which is closer to the intense-competition scenario assumed by the theoretical model.Source: §A.1 · p. 14
A.2 Details of the experimental exploration of other design choices
This section explores whether added design complexity can yield significant performance gains over NegotiaToR’s simplicity-oriented design. The exploration focuses mainly on the parallel network topology.
A.2 Details of the experimental exploration of other design choices
- The exploration tests whether added complexity yields significant performance gains over NegotiaToR’s simplicity-oriented design.The design prioritizes practicality through a minimalist approach.Source: §A.2 · p. 14
- The explored choices are iterative scheduling, traffic-aware selective relay, informative requests, and stateful scheduling.The section reports that these options may not provide proportionate performance gains.Source: §A.2 · p. 14
- Results are shown for the parallel network topology, except traffic-aware selective relay, which is designed for thin-clos; simulations otherwise use the §4.1 defaults.The section describes the exploration as design choices examined in §3.5.Source: §A.2 · p. 14
A.2.1 Iterative NegotiaToR Matching.
The iterative version of NegotiaToR Matching can improve matching efficiency, but added scheduling delay and stale traffic information may offset that gain. Simulations find iteration worsens FCT and does not improve goodput over non-iterative scheduling with 2× speedup.
A.2.1 Iterative NegotiaToR Matching
- Iteration may use unmatched ports to improve matching efficiency, but longer scheduling delay can worsen FCT and leave traffic-demand information outdated.Outdated demand may waste links; the resulting FCT and goodput may be adversely affected.Source: §A.2.1 · p. 14
- Each additional iteration sends another request after deriving the accept, with unmatched-port indices, and adds three epochs of scheduling delay.The iterative design repeats requests to destinations before data transmission.Source: §A.2.1 · p. 14
- The comparison tests 3 and 5 iteration rounds without speedup against the non-iterative version with 2× speedup on the parallel network topology.Scheduling-message sizes are held equal, although iteration would require extra bits for port-occupation information.Source: §A.2.1 · pp. 14–15
- At all loads, iterative scheduling worsens FCT, and its goodput is equal to or below the non-iterative version with 2× speedup.Goodput decreases with more iterations; the results indicate that increasing speedup is more effective than iteration for compensating for low matching efficiency.Source: §A.2.1 · p. 15, Figure 15 · p. 15
A.2.2 Traffic-aware selective relay.
Traffic-aware selective relay is explored as a way to improve link utilization in thin-clos when empty links are available. Simulations show little goodput improvement, so NegotiaToR does not employ data relay.
A.2.2 Traffic-aware selective relay
- NegotiaToR uses one-hop paths, avoiding traffic-oblivious relay’s doubled data volume, extra hops, and need for congestion control at intermediate nodes.The paper associates these relay costs with possible goodput and mice-flow FCT damage under heavy loads.Source: §A.2.2 · p. 15
- Relay can add paths and enable simultaneous transmission over multiple ports in thin-clos, unlike the parallel topology where each ToR pair can use all ports directly.This potential benefit motivates selective relay when empty links are available.Source: §A.2.2 · p. 15
- The scheme considers relaying elephant flows through lightly loaded ToRs while limiting flow size and avoiding intermediate links that conflict with high-volume direct traffic.The traffic-aware design also uses congestion control at the intermediate ToR to prevent buffer overflow.Source: §A.2.2 · p. 15
- Mice data stays on the highest-priority direct path; elephant data is relayed only above a volume threshold, and candidates grant requests only with spare queue capacity and no high-volume direct-traffic conflict.The ACCEPT step prioritizes direct traffic over relayed traffic.Source: §A.2.2 · p. 15
- With traffic-aware selective relay, FCT is barely affected and goodput improves only slightly, so NegotiaToR does not employ data relay.At light loads, 2× speedup already gives near-optimal goodput; at heavy loads, saturated links leave little benefit and the scheme usually turns relay off.Source: §A.2.2 · p. 16, Table 3 · p. 16
A.2.3 Informative requests.
Informative requests add queue-size or head-of-line waiting-time information to scheduling, aiming to improve goodput or tail FCT. Simulations find limited benefits overall, favoring binary requests and scheduling-delay bypass over the added complexity.
A.2.3 Informative requests.
- Queue-size requests prioritize source-destination pairs with more queued demand, while HoL waiting-time requests prioritize pairs with longer delays to potentially reduce starvation and tail FCT.Both approaches require added complexity, including queue-size access, packet-delay logging, or sorting.Source: §A.2.3 · p. 16
- A weighted HoL waiting time gives the highest-priority queue greater weight; simulations find best performance with a small, non-zero α such as 0.001.This setting supports prompt scheduling of mice-flow pairs while still indicating elephant-flow transmission needs.Source: §A.2.3 · p. 16
- Binary requests with scheduling-delay bypass are sufficient for high performance; informative-request benefits do not justify their additional complexity.The HoL-delay approach reduces FCT by 29.9% at 100% load but yields minor-to-zero improvements at other loads; data-size requests slightly improve goodput while adversely affecting FCT.Source: §A.2.3 · p. 16
A.2.4 Stateful scheduling.
The appendix examines stateful scheduling, an alternative scheduling algorithm, workload-level bandwidth behavior, and link failures. The observations report broadly unchanged performance with stateful scheduling, advantages for NegotiaToR in the tested workloads, and intermittent transmission suspension when failures disrupt scheduling messages.
A.2.4 Stateful scheduling.
- With stateful scheduling, overall performance roughly remains the same, so sources continue sending requests when they have pending data.Destination ToRs track incoming demands and grant only when the matrix indicates pending data, preventing unnecessary scheduling.Source: §A.2.4 · p. 16
A.3 Micro-observation of NegotiaToR’s performance under incast and all-to-all workloads
- In degree-15 incast, NegotiaToR sends data during predefined phases and the destination receives it shortly after injection; its behavior is identical on the parallel and thin-clos topologies.Traffic-oblivious designs have a longer interval before destination reception because data must first be relayed through a third ToR.Source: Figure 17 · p. 17, §A.3 · p. 17
A.2.5 Other design choices.
- ProjecToR scheduling performs worse than NegotiaToR in both FCT and goodput; additional iterations may improve goodput but increase scheduling delay and mice-flow FCT.The comparison keeps NegotiaToR’s incast bypass and mice-priority queue, and uses one iteration with bundle size equal to one epoch’s data.Source: §A.2.5 · p. 17
A.3 Micro-observation of NegotiaToR’s performance under incast and all-to-all workloads
- For 30 KB all-to-all flows, NegotiaToR’s receiver gets desired traffic continuously at high bandwidth, while traffic-oblivious designs also receive intermediate traffic that competes for receiver bandwidth.The sampled flows are injected at 10 μs.Source: Figure 18 · p. 17, §A.3 · p. 17
A.4 Micro-observation of NegotiaToR’s behavior under link failures
- Link failures can reduce receiver bandwidth to the capacity of remaining links, and lost scheduling messages can create epochs with zero bandwidth occupation.Changing the predefined phase’s round-robin rule sends scheduling messages through different links over time, mitigating reliance on a single failed link and preventing continuous zero occupation.Source: Figure 19 · p. 17, §A.4 · p. 18