Datacenter Traffic Control: Understanding Techniques and Trade-offs

Mohammad Noormohammadpour; Cauligi S. Raghavendra

Source-linked AI summary

Datacenter Traffic Control: Understanding Techniques and Trade-offs

Mohammad Noormohammadpour, Cauligi S. Raghavendra

arXiv:1712.03530v1Dec 10, 2017cs.NIcs.DCcs.PFeess.SY

TL;DR

Datacenter traffic control must accommodate diverse, application-dependent workloads while efficiently sharing network resources. This tutorial synthesizes control techniques and trade-offs, concluding that most proposals remain far from industry adoption and that inter-datacenter communication needs further attention.

Problem
Application-dependent traffic has diverse characteristics, making general datacenter traffic-control conclusions difficult despite the need to use resources efficiently and share them fairly.
Method
The paper reviews datacenter traffic-control challenges, mechanisms, proposed solutions, and trade-offs across intra- and inter-datacenter networking.
Results
Most traffic-control proposals remain far from industry adoption when complexity, performance, and cost are considered together, while inter-datacenter communication remains an evolving research area.
Takeaways & Limitations
Further research should pursue higher-performance datacenter traffic-control solutions with equal or lower cost and low complexity, alongside continued work on inter-datacenter communication.

Abstract

from arXiv · show

Datacenters provide cost-effective and flexible access to scalable compute and storage resources necessary for today's cloud computing needs. A typical datacenter is made up of thousands of servers connected with a large network and usually managed by one operator. To provide quality access to the variety of applications and services hosted on datacenters and maximize performance, it deems necessary to use datacenter networks effectively and efficiently. Datacenter traffic is often a mix of several classes with different priorities and requirements. This includes user-generated interactive traffic, traffic with deadlines, and long-running traffic. To this end, custom transport protocols and traffic management techniques have been developed to improve datacenter network performance. In this tutorial paper, we review the general architecture of datacenter networks, various topologies proposed for them, their traffic properties, general traffic control challenges in datacenters and general traffic control objectives. The purpose of this paper is to bring out the important characteristics of traffic control in datacenters and not to survey all existing solutions (as it is virtually impossible due to massive body of existing research). We hope to provide readers with a wide range of options and factors while considering a variety of traffic control mechanisms. We discuss various characteristics of datacenter traffic control including management schemes, transmission control, traffic shaping, prioritization, load balancing, multipathing, and traffic scheduling. Next, we point to several open challenges as well as new and interesting networking paradigms. At the end of this paper, we briefly review inter-datacenter networks that connect geographically dispersed datacenters which have been receiving increasing attention recently and pose interesting and novel research problems.

I. INTRODUCTION … 1) Fat-Tree:

The paper introduces datacenter networks, their diverse application-driven traffic, and the need to manage performance, scalability, cost, and complexity. It then surveys traffic-control concepts and network topologies, emphasizing trade-offs and Fat-Tree characteristics.

I. INTRODUCTION: Datacenters support diverse scalable services, while application workloads create varied communication patterns and substantial internal traffic.Examples include web services, cache followers, file stores, key-value stores, data mining, search indexing, and web search; scatter-gather applications can generate substantial internal traffic.
I. INTRODUCTION: Datacenter networks seek high bandwidth and low latency, but topology choices must balance resiliency, expansion, capacity, and deployment cost.Typical latency is up to hundreds of microseconds, while full bisection bandwidth can be costly and unnecessary for smaller organizations.
II. RELATED WORKS: The tutorial surveys traffic-control concepts across transport protocols and highlights trade-offs involving complexity, performance, and cost.It differs from prior work by covering multiple traffic-control aspects rather than focusing on a single problem or technique.
III. DATACENTER NETWORKS: The datacenter-network review covers proposed topologies, general traffic properties, traffic-control challenges, and objectives of operators and tenants.These objectives are presented as concerns of multiple parties involved in using datacenter networks.
A. Topologies: The topology review examines physical datacenter designs proposed or used in literature and illustrates their inter-ToR network connections.The examples are considered in testbeds or simulations.
1) Fat-Tree:: Fat-Tree is a multi-rooted, folded-Clos topology that uses many inexpensive switches to provide high bisection bandwidth and support large clusters.Increasing the number of roots reduces the over-subscription ratio when link capacity is fixed.
1) Fat-Tree:: Effectively using Fat-Tree paths can require complex switch routing, while Leaf-Spine expands capacity and ports by adding spine or leaf switches.Portland is an example of custom routing and forwarding designed to improve fault tolerance, scalability, and management.

2) Leaf-Spine: … B. Traffic Properties

Datacenter topologies trade off capacity, structure, scalability, fault tolerance, and implementation requirements, while traffic properties vary substantially with applications and deployment. Measurements show predominantly short flows alongside bursts, application-specific size and duration distributions, and workload-dependent rack locality.

2) Leaf-Spine; 3) VL2; 4) JellyFish: VL2 uses a folded-Clos network with higher-capacity switch-to-switch links, while JellyFish randomly connects ToRs according to port-availability rules.VL2’s bipartite aggregation structure reduces the number of cables needed between aggregation and core layers.
5) DCell; 6) BCube; 7) Xpander: DCell supports incremental expansion and failure-aware near-shortest-path routing; BCube uses server forwarding and source routing; Xpander provides structured expander-based connectivity and higher throughput during expansion.BCube’s software forwarding can add CPU overhead, while Xpander improves implementation predictability over JellyFish and can use k-shortest paths with MPTCP.
B. Traffic Properties: Traffic characteristics—including flow arrivals, sizes, durations, and microbursts—depend strongly on applications, which can create both long-lived connections and short bursts.Web-search flows are generally smaller and shorter than batch-computing flows.
B. Traffic Properties: More than 50% of flows lasted under 100 ms, 80% under 10 seconds, and almost all under 100 seconds in a MapReduce-like query-processing cluster.
B. Traffic Properties: Packet arrivals at monitored ToR switches exhibited an ON/OFF pattern, with variable idle periods between bursts across 19 datacenters and diverse workloads.
B. Traffic Properties: Across 10 educational, enterprise, and cloud datacenters, over 40% of flows were below 1 KB, over 80% below 10 KB, and almost all below 10 MB.The reported number of active flows was below 10,000 per second per rack across all datacenters.
B. Traffic Properties: Facebook measurements show application-specific flow sizes and durations, with Cache flows lasting far longer than Hadoop or Web Server flows.Median durations were about 300 ms for Hadoop, 900 ms for Web Server, and almost 400 seconds for Cache; their tails were below 1000, about 200, and almost 800 seconds, respectively.
B. Traffic Properties: Rack-locality ratios depend on application placement and communication patterns, making locality relevant to cluster interconnection oversubscription decisions.

C. Traffic Control Challenges · 1) Unpredictable Traffic Matrix: · 2) Mix of various flow types/sizes:

Datacenter traffic is difficult to control because highly variable flow arrivals and sizes create an unpredictable traffic matrix. Applications also generate heterogeneous interactive, throughput-oriented, and deadline-constrained flows with distinct performance requirements.

1) Unpredictable Traffic Matrix:: Most datacenter flows are short, often only a few packets, while most bytes are delivered by large flows.Short flows are usually less than 1 MB.
1) Unpredictable Traffic Matrix:: Flow-size distributions are heavily skewed: 80% of flows are less than 10 KB and 99% are less than 100 MB.
1) Unpredictable Traffic Matrix:: A single machine’s median flow inter-arrival time ranges from 2 ms to 10 ms, equivalent to 100 to 500 flows per second.
1) Unpredictable Traffic Matrix:: High arrival rates dominated by short flows produce a fluctuating traffic matrix that complicates long-timescale traffic engineering and capacity planning.
2) Mix of various flow types/sizes:: Datacenter traffic combines flow types and sizes whose application-determined priorities and requirements can guide transport-protocol design, although some sizes are unknown at initiation.
2) Mix of various flow types/sizes:: Interactive flows are typically short, latency-sensitive, and high-priority; examples include queries of 2 to 20 KB and messages of 100 KB to 1 MB.
2) Mix of various flow types/sizes:: Throughput-oriented flows tolerate delay but require consistent bandwidth, spanning 1 MB to 100 MB transfers and long-running inter-datacenter data movements.
2) Mix of various flow types/sizes:: Deadline flows must finish before soft or hard deadlines, with interactive scatter-gather queries usually requiring completion within less than 300 ms.Hard-deadline flows lose all value after expiry, whereas soft-deadline flows retain decreasing value.

3) Traffic Burstiness: … 1) Minimizing Flow Completion Times:

The paper identifies burstiness, packet reordering, shared-resource contention, incast, and outcast as datacenter networking challenges, while traffic control must address competing stakeholder objectives and minimize flow completion times.

3) Traffic Burstiness:: Bursty traffic increases packet loss and queuing delay, decreases throughput, and can raise queue occupancy, flow completion times, and packet drops.Burstiness is associated with more frequent packet losses at the network edge.
4) Packet Reordering:: At high rates, significant packet reordering can raise CPU utilization to 100% and limit server interface link utilization to 70%.Reordering can also increase receiver memory and CPU utilization, latency, and erroneous fast retransmissions.
5) Performance Isolation:: Performance isolation is necessary to prevent tenants from unfairly obtaining shared resources or causing disruptions.Effective isolation requires per-tenant or per-user controls for buffers, bandwidth, and communication-related computational and memory overheads.
6) The Incast Problem:: Incast occurs when many end-hosts simultaneously send to one destination, creating a destination-link bottleneck with queue buildup, large delays, and packet drops.Search and MapReduce applications can produce this partition-aggregate, many-to-one traffic pattern.
7) The Outcast Problem:: Outcast, or port blackout, results from synchronized packet drops under specific contention and TailDrop conditions, eventually causing unfairness.In tree-based topologies and partition-aggregate communication, some input ports may receive less average throughput.
D. Traffic Control Objectives:: Datacenter traffic control must account for operators, tenants, and end-users whose objectives differ across resource use, service provision, and fairness.Operators seek high resource utilization and greater service volume, while tenants seek resources aligned with their service level agreements.
1) Minimizing Flow Completion Times:: Flow Completion Time is the interval from flow initiation to completion and can affect user quality of experience and service, with queuing and packet loss as major determinants.Retransmitting lost segments can significantly increase latency because identifying and retransmitting lost data takes time.

2) Minimizing Deadline Miss Rate or Lateness: … A. Distributed

The paper frames datacenter traffic control around meeting deadlines or minimizing lateness, maximizing utilization, fairness, and energy efficiency. Management can be distributed, centralized, or hybrid; distributed designs emphasize scalable coordination through end hosts and, when useful, network elements.

2) Minimizing Deadline Miss Rate or Lateness:: Deadline-sensitive applications require timely delivery, while minimizing lateness can matter when post-deadline delivery retains value.Service quality decreases as the fraction of missed deadlines increases.
3) Maximizing Utilization:: Better network-bandwidth utilization can reduce provisioning costs, increase revenues, and support more tenants or better service for existing ones.Utilization depends partly on topology and design parameters and partly on the traffic-control scheme.
4) Fairness:: Fairness allocates shared bandwidth and buffer resources according to tenant SLAs, mitigating starvation and malicious behavior.Fairness is relevant because many flows share datacenter resources in multi-tenant environments.
4) Fairness:: Networking fairness includes max-min, proportional, and balanced fairness; max-min fairness, which maximizes the minimum share, is most widely used.The paper focuses on fairness in network bandwidth, although fairness can span bandwidth, CPU cycles, and memory.
4) Fairness:: Fairness policies should select appropriate entities, because strict fairness across flows can increase missed deadlines and produce sub-optimal flow completion times.Fairness may be applied across flow groups rather than individual flows to prevent tenants from gaining resources by creating many flows.
4) Fairness:: Datacenter operators may also pursue energy efficiency because networks contain many equipment items and average load can remain below peak as computational and storage loads vary.The passage identifies reducing energy costs through greater energy efficiency as an additional objective.
IV. DATACENTER TRAFFIC CONTROL MANAGEMENT: Traffic-control management ranges from fully distributed to completely centralized, with hybrid schemes forming a third main approach.These approaches require some coordination across network elements.
A. Distributed: Distributed control is generally more reliable and scalable, can run at end hosts or switches, and includes end-host methods such as SFS for receiver-side incast control.SFS uses ACK generation to regulate traffic toward receivers and schedules higher-priority flows; end-host implementations are preferred when they avoid switch changes and are more scalable because each server handles its own traffic.

1) Benefits: … C. Hybrid

Datacenter traffic control spans distributed, centralized, and hybrid management schemes, each trading local simplicity, global coordination, scalability, reliability, and performance. Hybrid designs combine distributed fine-grained control with centrally calculated parameters to balance these trade-offs.

1) Benefits:: Distributed control uses local network and flow information, which can produce locally optimal decisions that reduce overall network utility.For example, endpoints maximizing their own throughput may disadvantage endpoints carrying more critical traffic.
2) Drawbacks:: Distributed schemes can improve reliability and scalability but may provide less globally coordinated performance.Their limited network-wide visibility makes enforcing new global policies more difficult.
B. Centralized: Centralized control coordinates transmissions using global topology, resource, switch-state, and end-host-demand information.Schedulers can allocate resources across time and links while accounting for flow sizes, deadlines, priorities, queue status, and link capacities.
2) Drawbacks:: Centralized designs face single-point-of-failure, hotspot, information-collection, computational, allocation, and scheduling overheads.Scheduling can also delay flow initiation, while hierarchical designs may improve scalability for larger networks.
1) Benefits:: Central management can improve performance by directing traffic across available paths and prioritizing flows using a global network view.Relevant information includes edge utilization and health, as well as flow size, deadline, and priority.
C. Hybrid: Hybrid control combines distributed reliability and scalability with performance gains from global management through centrally calculated parameters.Central components can provide coarse-grained decisions while distributed components apply fine-grained control, reducing the weaknesses of either scheme.
C. Hybrid: Hybrid systems can divide responsibilities hierarchically, such as centrally calculating per-tenant aggregate bandwidth while controlling transmissions in a distributed manner.Examples include OTCP, Fibbing, Hedera, and Mahout; these systems use centralized measurements or end-host and switch monitoring to improve scalability.

1) Benefits: … B. Traffic Shaping

Datacenter traffic control techniques span transmission control and traffic shaping, with trade-offs involving rate precision, burstiness, flexibility, isolation, and control centralization. Traffic shaping improves resource use by enforcing profiles and policies, while transmission mechanisms regulate sender rates through windows, explicit rates, or quotas.

2) Drawbacks:: Centralized control can create a single point of failure, whereas distributed components operate locally and may produce less globally optimal solutions.A distributed approach can reduce failure impact compared with a fully centralized scheme, but complexity may increase.
V. DATACENTER TRAFFIC CONTROL TECHNIQUES: Traffic control schemes involve many design parameters and approaches, organized into major techniques that relate to datacenter challenges and objectives.The paper presents high-level breakdowns and relationships among challenges, objectives, and techniques.
A. Transmission Control: Window-based transmission control regulates outstanding bytes and implicitly controls rate, but larger windows can increase average transmission rate.The sender window determines how much data can be transmitted before acknowledgements are received.
A. Transmission Control: Window-based control offers only coarse-grained rate control, causing considerable variations and potentially bursty traffic as senders release packet batches.A sender may transmit a full window, wait for acknowledgements, and then send another full window.
A. Transmission Control: Explicit rate control schedules outstanding packets to achieve target bit rates, reducing burstiness and supporting reservation-based bandwidth control over shared links.Rate-based mechanisms decrease transmission-rate variation at senders.
A. Transmission Control: Explicit rate control can run in hardware or software: hardware provides more precise rates, while software offers greater flexibility and supports more flows and complex policies.Software rate control may still create occasional bursts because precise packet scheduling is difficult in operating-system kernels.
A. Transmission Control: Token-based transmission control adds quotas so senders transmit only according to assigned credit, including quotas based on receiver congestion.This complementary approach is also called pull-based or credit-based control and can address incast.
B. Traffic Shaping: Traffic shaping enforces required profiles and policies to reduce contention, prevent flows from hogging resources, provide isolation and guarantees, and support scheduled sending rates.These functions are particularly relevant in multi-user cloud environments and resource scheduling.

1) Rate Limiting: · 2) Packet Pacing:

Rate limiting controls average transmission rates and burstiness through token-based or other mechanisms, while packet pacing spaces packets to reduce burst-related performance problems. Both techniques involve trade-offs among precision, computational or hardware complexity, traffic classes, and network conditions.

1) Rate Limiting:: Rate limiting commonly uses a Token Bucket filter, which bounds average transmission rate by token generation rate and caps accumulated tokens to limit burstiness.Packets are sent only when tokens are available; idle periods can accumulate tokens up to the cap.
1) Rate Limiting:: Software rate limiting in kernels or hypervisors can be imprecise and computationally intensive at high bandwidths or with many flows, while batching further reduces scheduling precision.Linux Classful Queuing Disciplines provide coarse-grained rate control, and NIC scheduling can reduce CPU load and traffic burstiness.
1) Rate Limiting:: Hardware rate limiting requires multiple queues and traffic classes, whereas hybrid designs use host classification with NIC scheduling to support many priority classes while limiting hardware complexity.NicPic stores labeled packets in RAM, then uses DMA and hardware rate limiters to schedule them.
1) Rate Limiting:: Application-layer rate limiting is simple but requires application changes, offers limited precision, and can create bursty wire traffic when data is buffered before transmission.Applications control the volume handed to the transport layer over time but not necessarily how data is ultimately sent.
2) Packet Pacing:: Packet pacing spaces consecutive packets to prevent back-to-back arrivals and reduce traffic burstiness.The technique is a form of traffic shaping that distributes transmissions over time.
2) Packet Pacing:: Burstiness can overload switch buffers, cause consecutive drops, increase average packet latency, and produce transmission-rate oscillations that hinder bandwidth allocation.These effects arise from long bursts creating longer queues and unstable rates.
2) Packet Pacing:: Pacing can reduce queuing delay and improve performance when combined with congestion signals, but should target long-running throughput-oriented flows rather than short latency-sensitive flows.Its benefit depends on the network bandwidth-delay product, buffer size, and number of flows; sharing a buffer among many flows can reduce effectiveness.
2) Packet Pacing:: Hardware pacing can be more effective than software pacing because its higher scheduling precision matters at high rates, although pacing can also occur in drivers, kernels, hypervisors, or network-edge queues.Software pacing may be overridden by NIC offloading, while QLBP uses edge-queue length to determine when to send the next packet.

C. Prioritization · D. Load Balancing

The paper presents prioritization as a way to differentiate traffic by flow size, deadlines, or control importance, while load balancing distributes traffic across redundant paths to reduce contention and improve utilization. Both mechanisms involve trade-offs involving starvation, reordering, congestion, and responsiveness.

C. Prioritization: Prioritization improves performance by treating flows according to priority, using switch queues or sender rate control.Priorities may be assigned by flow size, deadlines, or control importance.
C. Prioritization: Dynamic prioritization starts packets at high priority and demotes flows as more packets are observed when sizes are unknown.DPP uses express and normal queues, while up to 8 queues can provide most mean-FCT reduction benefits.
C. Prioritization: Priorities can be assigned in switches or tagged by end hosts, trading simpler hosts against greater switch state and processing demands.IEEE 802.1Q provides 8 priority levels, while other approaches include DiffServ and custom queuing.
C. Prioritization: Strict prioritization can starve lower-priority flows, so weighted queuing or aging can preserve their progress while limiting effects on critical traffic.DAQ uses weighted round-robin between long and short flows.
D. Load Balancing: Load balancing distributes traffic across redundant paths to reduce contention and improve utilization, using static criteria or adaptive responses to network load.ECMP statically distributes traffic across equal-cost paths, whereas adaptive methods can use queue occupancies and react to hot spots.
D. Load Balancing: Load balancing may operate per packet, packet group, or flow, trading finer distribution against packet reordering and congestion from poorly placed large flows.Per-flow balancing minimizes reordering but can provide limited benefit for many throughput-oriented flows; rerouting can introduce instability.
D. Load Balancing: Flowlet and flowcell scheduling balance load while controlling reordering through group size or timeout choices.Flowlet timeout values trade finer balancing against reordering, while flowcell thresholds include 10 KB at ToR switches and 64 KB at the hypervisor layer.

1) Data and Task Placement: … E. Multipathing

Datacenter performance depends on coordinating data and task placement, routing and forwarding, failure-aware load balancing, and multipathing. These mechanisms expand path and placement choices but introduce scalability, setup, buffering, and heterogeneous-path trade-offs.

1) Data and Task Placement:: Data placement determines which replicas or erasure-coded pieces can serve computations, while distributing data across racks and servers expands load-balancing options.Ceph distributes copies according to popularity and spreads large or write-heavy directories to reduce hot-spots; HDFS also considers network topology.
1) Data and Task Placement:: Placing computation near its data can reduce network usage, contention, and queuing, while considering flow-scheduling policies can improve overall task completion times.Relevant policies include FCFS, SRPT, and Fair Sharing.
2) Routing and Forwarding:: Switch forwarding uses FIB rules installed proactively or reactively, trading potentially unused rule scale against setup-time overhead.Rules may be installed directly or through routing protocols that calculate and install them.
2) Routing and Forwarding:: Load balancing can use VLAN-based Layer 2 spreading, Layer 3 routing with multiple next hops such as ECMP, or centralized controllers that install forwarding state.VLAN-based balancing has limited scalability because it creates large broadcast domains; centralized approaches can calculate FIBs and account for path diversity.
2) Routing and Forwarding:: Source routing moves forwarding information into packets, simplifying network state dissemination but requiring end-host topology awareness and network-status mechanisms.PSSR encodes outgoing port numbers rather than addresses, decoupling addressing from forwarding.
3) Effect of Failures:: Network failures reduce capacity but rarely eliminate reachability because datacenter networks usually provide substantial redundancy.Failures can affect switches, ports, links, or other network elements despite the use of high-quality equipment.
3) Effect of Failures:: Capacity asymmetries across paths motivate weighted and adaptive load-balancing methods, including WCMP, CONGA, HULA, Presto, LetFlow, DRILL, and Hermes.WCMP extends ECMP by assigning path weights proportional to capacity and using those weights to determine hash entries per outgoing port.
E. Multipathing: Multipathing splits flows across paths to improve throughput and failure reliability, but receiver buffering, sub-flow setup, and path heterogeneity can limit its benefits.Buffering scales with aggregate path throughput times the longest-path latency; multipathing may be useful for long background flows, whereas short flows can incur setup and teardown overhead, and highly utilized or mismatched paths may yield little or negative throughput gain.

F. Scheduling … 3) Deadline-Awareness:

The paper frames datacenter scheduling as a computationally expensive optimization problem balancing throughput, latency, fairness, bandwidth guarantees, and deadline compliance. It reviews reservation, redundancy, deadline-aware scheduling, and common scheduling disciplines as mechanisms for managing heterogeneous flows.

F. Scheduling: Scheduling optimizes utilization, fairness, or latency while maximizing throughput for bandwidth-hungry flows and minimizing flow completion time for latency-sensitive flows.Techniques also reduce FCT and provide bandwidth guarantees to long-running flows.
F. Scheduling: Mixing flow types creates a complex scheduling optimization problem that is generally computationally expensive to solve.The paper summarizes scheduling techniques in Table X.
1) Reservation:: Reservation checks resource availability and allocates resources before transmission to improve bandwidth guarantees and prevent congestion spots.Requests may be processed centrally or in a distributed fashion.
1) Reservation:: Centralized reservation lets end-hosts report demands to a scheduler that allocates transmission slots or rates; TDMA uses a fabric manager to assign contention-less slots.Examples include TDMA, FastPass, FlowTune, and TAPS.
1) Reservation:: Distributed reservation considers requested rates and available bandwidth to determine a flow’s rate and path, with allocation performed at switches or by receivers.RCP and PDQ allocate rates at switches, while receivers can schedule multiple incoming flows.
2) Redundancy:: Replicating flows and using the fastest responding replica can reduce tail latency, while selectively reissuing slow requests limits redundancy overhead.The number of redundant requests should balance resource usage and response time because only a small portion of flows are usually laggards.
3) Deadline-Awareness:: Deadline-aware scheduling minimizes deadline miss rate and may terminate hard-deadline flows early when their deadlines cannot be met.D2TCP, D3, PDQ, and MCP regulate rates according to flow deadlines.
3) Deadline-Awareness:: Finishing deadline flows earlier than necessary can worsen latency-sensitive traffic’s FCT, motivating optimization across mixed-flow scenarios and established scheduling disciplines.FCFS provides bounded lateness for deadline flows and is close to optimal for tail completion times under lighttailed flow-size distributions; PS shares resources equally.

4) Disciplines: … VI. OPEN CHALLENGES

The paper reviews scheduling disciplines and related traffic-control mechanisms, then identifies open challenges arising from workload heterogeneity, path selection, control-loop delays, and conflicting objectives.

4) Disciplines:: RCP enforces processor sharing, while PDQ prioritizes EDF before SJF to reduce deadline misses and then flow completion time.FastPass targets user-level max-min fairness; pFabric and SFS use shortest-remaining-flow-first scheduling, while PIAS uses dynamic priority assignment.
4) Disciplines:: Industry switches commonly provide FIFO queues implementing FCFS, while priority queues can approximate SRPT by assigning smaller flows higher priority.
5) Preemption:: Online arrivals can create race conditions and priority inversion; SJF may perform poorly when short flows arrive after a large flow, motivating preemptive SRPT.
6) Jittering:: Jittering adds random application-layer delays to simultaneous fetch requests, desynchronizing traffic and reducing peaks that can cause congestion, packet drops, and latency.
7) ACK Control:: Receivers can control transmission by delaying or withholding ACKs, including pausing acknowledgments for low-priority flows when higher-priority traffic arrives.
VI. OPEN CHALLENGES: Traffic-control optimization is often computationally expensive and difficult to enforce because of complex constraints, limited hardware support, slow software response, failures, and errors.
VI. OPEN CHALLENGES: Mixed workloads require balancing differing goals such as search latency, backup throughput, and deadline requirements, while load balancing must trade bandwidth utilization against packet reordering.Per-packet balancing provides finer balancing but generally causes significant reordering; per-flow balancing avoids reordering but may leave capacity underused.
VI. OPEN CHALLENGES: Control-loop delay limits simultaneous achievement of high throughput and low latency, while fairness, latency, throughput, and deadline objectives can conflict and require utility-based optimization.The paper identifies maximizing utility across latency, utilization, deadline miss rate, lateness, and fairness as an open problem.

VII. RELATED PARADIGMS … D. Userspace Packet Processing

Datacenter networking paradigms increasingly provide operators with flexible, controllable mechanisms for performance optimization, spanning programmable control and data planes, advanced NIC offloads, and userspace packet processing. These approaches enable custom policies, faster hardware updates, high-rate transport operations, and reduced packet-processing overhead.

VII. RELATED PARADIGMS: Networking paradigms have made datacenter networks more flexible and controllable, expanding operators’ opportunities for performance optimization.The paper reviews several paradigms that affect datacenter network design and operation.
A. Programmable Forwarding Planes: Programmable forwarding planes support custom network-control policies and algorithms, including centralized management through controllers that account for policies and resource constraints.This approach is identified as part of Software Defined Networking (SDN).
A. Programmable Forwarding Planes: OpenFlow’s open interface enables switches from different vendors to be operated uniformly and simplifies network updates by requiring patches only to controllers.Multiple controllers may each manage part of the network while coordinating with one another.
B. Programmable Data Planes: Programmable data planes address fixed ASIC functionality by allowing switch packet-processing functions to change, reducing the delay and unpredictability of introducing new functions.PISA provides a realization path, while P4 is a vendor-independent and protocol-independent language for programming PISA switches.
C. Advanced NICs: Advanced NICs offload complex transport and networking operations, including cryptography, quality of service, congestion control, storage acceleration, erasure coding, and policy enforcement.These operations can run at datacenter line rates of 40 Gbps and more, avoiding significant OS CPU overhead and additional communication latency.
D. Userspace Packet Processing: By default, packets traverse the Operating System networking stack, with NIC interrupts invoking OS routines and batching reducing CPU utilization at high rates.Interrupt Moderation is given as an example of batching packet processing.
D. Userspace Packet Processing: Kernel bypass modules replace interrupt-driven OS processing with polling to reduce packet-processing latency and increase throughput.Netmap, VPP, and DPDK have been shown to reduce the cycles required to process a packet by up to 20× on average.
D. Userspace Packet Processing: Userspace networking stacks and applications use kernel bypass and polling for custom packet processing and faster RPCs, including Sandstorm, mTCP, SoftNIC, RAMCloud, FastPass, and NDP.Sandstorm and mTCP implement TCP in userspace, while SoftNIC enables custom NIC features in software.

E. Lossless Ethernet and RDMA · VIII. BROADER PERSPECTIVE · A. Private Dedicated Inter-datacenter Networks

The paper contrasts TCP with RDMA and explains the lossless Ethernet mechanisms and deployment options needed to support RDMA. It then broadens the perspective to inter-datacenter traffic growth, private dedicated networks, and centralized bandwidth allocation, routing, and rate control.

E. Lossless Ethernet and RDMA: RDMA bypasses the OS networking stack on NICs, offering higher bandwidth, lower latency, and lower CPU utilization than TCP.RDMA can also offload large datasets to nearby machines instead of using pagefiles.
E. Lossless Ethernet and RDMA: RDMA requires a lossless underlying network, while Lossless Ethernet provides Layer 2 per-hop flow control through backpressure and PAUSE messages.PAUSE messages are sent to previous hops when a network buffer is full.
E. Lossless Ethernet and RDMA: RDMA can use RoCE across a Layer 2 domain, RoCEv2 over IP and UDP across Layer 3 networks, or iWARP over IP and TCP with lossless end-to-end transport on NICs.iWARP does not require a lossless infrastructure.
E. Lossless Ethernet and RDMA: Lossless Ethernet can cause Layer 2 head-of-line blocking, unfairness, and deadlocks because pausing operates per port or class and may create circular dependencies.A flow can overflow a port and block other flows using that port.
E. Lossless Ethernet and RDMA: QCN reduces PAUSE messages by notifying senders before buffers fill, while TCP Bolt and DCQCN use ECN-like marking over RoCEv2 to reduce buffer occupancy and minimize PAUSE signals.TCP Bolt also creates edge disjoint spanning trees with different PFC classes to prevent cyclic dependencies; TIMELY uses delay to detect increased buffer occupancy.
VIII. BROADER PERSPECTIVE: Inter-datacenter traffic includes user interaction and business-internal processing or migration, with business-internal traffic growing much faster than user-generated traffic.Multiple datacenters support distributed applications by improving availability, fault tolerance, and data availability.
VIII. BROADER PERSPECTIVE: Large operators use private dedicated inter-datacenter networks, including Google B4, Microsoft Global WAN, and Facebook Express Backbone, creating opportunities for network-resource optimization.These networks use long-haul optical connections, including private dark fiber in Microsoft’s case.
A. Private Dedicated Inter-datacenter Networks: Bandwidth allocation over private networks commonly reserves capacity or priority for latency-sensitive flows while allocating bandwidth per transfer to long-running flows.SDWAN can centrally manage transmission and routing according to transfer requirements, network status, and topology; a Traffic Engineering Server calculates rates and routes.

B. Research Directions … IX. CONCLUSIONS

The paper identifies inter-datacenter communication as an evolving research area, emphasizing faster global optimization, new objectives for one-to-many transfers, and failure-aware routing. It concludes that datacenter traffic control remains necessary for efficient resource use and fair sharing, while many proposals remain far from industry adoption.

B. Research Directions: Research directions cover inter-datacenter communication, including global optimization, one-to-many transfers, and failure-aware routing.The paper presents these as areas requiring further attention.
1) Inter-datacenter Global Rate-allocation and Routing:: Global rate-allocation and routing research targets objectives such as utilization maximization, deadline satisfaction, and admission control.Examples include B4, SWAN, Tempus, and Amoeba.
1) Inter-datacenter Global Rate-allocation and Routing:: Complex optimization problems can be computationally expensive and slow, increasing scheduling latency as transfers arrive.Future algorithms should compute rates and routes quickly while considering transfer completion times.
1) Inter-datacenter Global Rate-allocation and Routing:: Services distributed across datacenters near regional users seek better quality of experience by minimizing customer access latency.The passage gives CDNs caching objects for local viewers as an example.
2) Inter-datacenter One-to-Many Transfers:: One-to-many transfers motivate objectives such as maximizing the number of receivers that complete reception within a given period.Inter-datacenter multicast differs from intra-datacenter multicast because the network topologies differ substantially.
3) Inter-datacenter Failure-aware Routing:: Inter-datacenter links can have significantly different failure probabilities, potentially differing by as much as three orders of magnitude.Inter-datacenter traffic also contains classes with different quality-of-service requirements and priorities.
IX. CONCLUSIONS: Datacenter traffic control is necessary to use resources efficiently and share them fairly among tenants and applications with different services.These environments support high availability, reliability, and scalable computing at minimal costs.
IX. CONCLUSIONS: Despite substantial research, most traffic-control proposals remain far from industry adoption when complexity, performance, and cost are considered together.The paper reviews traffic-control challenges, elements, proposed solutions, and trade-offs, and presents a centralized architecture for inter-datacenter communication.