PaperlayerView exact revision on arXiv

Source-linked AI summary

CCAaaS: An Orchestration Platform for High-Velocity Design, Development, and Deployment of Data Center Transports at Meta

Prankur Gupta, Balasubramanian Madhavan, Miao Xu, Prashanth Kannan, Jie Meng, Neil Spring, Rajiv Krishnamurthy, Srikanth Sundaresan, Martin Lau, Theophilus A. Benson

arXiv:2610.33231v1cs.NI

TL;DR

Data-center networks must serve workloads with competing latency and throughput demands, while AI workloads accelerate the arrival of new traffic patterns. CCAaaS provides an eBPF-based ecosystem for developing, deploying, and observing coexisting CCAs; it has been used to develop five CCAs over five years and now carries over 97% of internal traffic.

  • Problem

    Data-center applications have competing latency and throughput requirements, and operators must develop and evolve coexisting CCAs as traffic patterns change.

    Source: §1 · p. 1

  • Method

    CCAaaS combines a modular eBPF-based BPF-TCP framework with CCA-specific deployment orchestration and lightweight observability.

    Source: §1 · p. 2

  • Results

    CCAaaS has been used to develop five CCAs over five years, and those CCAs carry over 97% of internal traffic; XRegion-CCA reduced cross-region service P99 latency by up to 20×.

    Source: §9.2 · p. 11, §12 · p. 12

  • Takeaways & Limitations

    The production experience yields design lessons for modular CCAs and identifies open challenges for the research community.

    Source: §12 · p. 12

    CCAaaS manages CCAs, but ensuring cross-CCA fairness is outside the system’s scope.

    Source: §11 · p. 12

Abstract

from arXiv · show

Today, data center networks run multiple congestion control algorithms (CCAs), each tailored to distinct traffic characteristics, from latency-sensitive intra-region RPCs to high-throughput cross-region AI workloads. The rapid evolution of AI training and inference further accelerates this trend by continually introducing new traffic characteristics that demand new CCAs. This paper introduces CCA-as-a-Service (CCAaaS), our eBPF-based ecosystem for the rapid design, deployment, and operation of production-grade CCAs at hyperscale. We present (1) a modular BPF-TCP framework and general lessons for designing modular CCAs; (2) a deployment and orchestration layer that provides disruption-free CCA updates, along with best practices for managing a zoo of CCAs at scale; and (3) a lightweight observability tool for CCAs and a blueprint for scaling CCA observability. CCAaaS manages over 97% of our server-to-server traffic using three active eBPF-based CCAs; two of the five CCAs we have built have since been deprecated. We show that our eBPF-centric approach to CCAs development delivers the same application-level performance (retransmissions, bandwidth, and tail latency) as their native in-kernel counterparts, and that our custom CCAs add negligible CPU and memory cost over their eBPF baselines. In terms of realized benefits, the CCA built with CCAaaS significantly improved network, and application metrics, for example, one of our CCAs reduced P99 latency for cross-region services by up to 20X.

1 Introduction

Data-center workloads impose competing latency and throughput demands, and a single congestion control algorithm (CCA) cannot meet all requirements. CCAaaS addresses the operational challenges of developing, deploying, and observing a growing portfolio of coexisting CCAs.

1 Introduction

  • CCA development must iterate quickly without datapath overhead, support safe incremental rollout, and handle rollback complications from persistent transport state.Some CCAs require both endpoints to be upgraded, while long-lived flows otherwise retain stale logic; negotiated ECN can persist after a CCA change.Source: §1 · pp. 1–2
  • CCAaaS is an eBPF-based ecosystem for rapid development and safe deployment of CCAs that coexist on production infrastructure.Its design addresses rollout safety, rollback correctness, debugging, and observability challenges overlooked by recent eBPF-based TCP CCA frameworks.Source: §1 · pp. 1–2
  • CCAaaS has been in production for over five years and has been used to develop five CCAs.The available passage cuts off before completing its statement about current deployment.Source: §1 · p. 2
  • DCNDelay-CCA reduced retransmissions for major services by 30–40%, while XRegion-CCA reduced cross-region service P99 latency by up to 20×.The paper also reports that custom eBPF-based CCAs manage over 97% of server-to-server traffic at Meta.Source: Figure 2 · p. 2
  • The system combines a modular BPF-TCP framework, CCA deployment and orchestration, lightweight CCALogger observability, and deployment lessons.The framework exposes modular hooks; orchestration supports safe rollback and disruption-free rollout; observability is co-designed with the framework.Source: §1 · p. 2

2 Motivation

Meta’s network serves traffic with markedly different characteristics, motivating multiple CCAs and a platform for operating them. The paper explains both the production need for coexistence and why eBPF fits the development and deployment constraints.

2.1 The Production Case for Multiple Simultaneous CCAs

  • Cross-region flows have a median RTT of ~32 ms versus ~70 µs for intra-region flows, motivating distinct CCA requirements.The section also describes different application demands and the challenges of safely coexisting CCAs on shared hosts, NICs, and switches.Source: §2.1 · pp. 2–3
  • Intra-region congestion often occurs at end hosts and NIC buffers, limiting ECN visibility, while cross-region delay changes can reflect traffic rerouting as well as congestion.These conditions motivate delay-based control intra-region and ECN-based control cross-region; loss-driven Cubic also reacts too late for long-lived AI workloads.Source: §2.1 · p. 3

2.2 Why eBPF for Congestion Control

  • eBPF runs in the kernel with kernel-level performance and allows programs to be loaded or replaced at runtime without host reboots.Direct kernel patches are slow to merge and require disruptive reboots; kernel modules complicate upgrades, while userspace stacks can require specialized hardware and sacrifice kernel TCP maturity.Source: §2.2 · pp. 3–4

3 CCAaaS Overview

CCAaaS separates CCA development and operation across a control plane and a data plane. The control plane manages eBPF object lifecycles and connection policies, while the data plane performs packet and event processing.

3 CCAaaS Overview

  • CCAaaS combines modular BPF-TCP hooks, deployment and orchestration for rollout and rollback, and lightweight CCA-specific observability.The system extends NetEdit with CCA-specific management primitives.Source: §3 · p. 4
  • The control plane selects, installs, and safely removes required eBPF objects, and installs policies governing CCA behavior for connections.It is not involved in per-packet or per-event processing.Source: §3 · p. 4
  • The data plane comprises eBPF objects, including maps and programs, used for per-packet and per-event processing.Source: §3 · p. 4

4 Modular CCA Framework

CCAaaS’s modular BPF-TCP framework gives developers control over core congestion control and selected cross-layer kernel behavior, while supporting state sharing and two tailored CCAs.

4 Modular CCA Framework

  • CCAaaS’s BPF-TCP interface provides control over the core congestion-control loop, cross-layer kernel functionality, and internal kernel configurations.The interface reflects lessons from refactoring and evolving it for production use cases.Source: §4 · p. 4

4.1 The BPF-TCP Hook Interface

  • BPF-TCP bundles CCA hook programs in struct_ops, atomically replacing callbacks and applying only one bundle to a connection.This avoids ordering and conflict-resolution among multiple CCAs on the same socket.Source: §4.1 · p. 4

4.1.1 Defining a New CCA

  • Developers implement a subset of struct_ops hooks, while operators specify connection locality and service scope for CCA deployment and configuration.CCAaaS installs the programs and their appropriate map configuration across the fleet.Source: §4.1.1 · pp. 4–5

4.1.2 BPF-TCP Interface Evolution

  • The interface evolved from core CCA hooks to expose precomputed per-RTT metrics, socket-level controls, and cross-layer controls such as TSO segmentation.The interface limits which CCAs are realizable: it lacks richer in-band telemetry and leaves some performance-critical socket functionality unexposed.Source: §4.1.2 · p. 5

4.2.1 Coordination Across Our Fleet

  • CCAaaS shares state through socket-indexed sk_storage for per-connection state and other map types for policy shared across programs.sk_storage lookups take O(1) time and cost about 25 ns in a production-host microbenchmark, versus about 1 µs for an LPM scope lookup.Source: §4.2.1 · pp. 5–6

4.2.2 CCA-Specific State-Sharing Edge Cases

  • Because per-connection CCA state exceeds the 104-byte icsk_ca_priv space, CCAaaS splits it across the internal CCA structure and sk_storage, adding one lookup per read.If scope-resolver maps are unavailable, an LPM lookup supplies and caches the result; this fallback is infrequent but expensive and may degrade application performance.Source: §4.2.2 · p. 6

4.3.1 DCNDelay-CCA: Delay-Based Control for Heterogeneous Data Centers

  • DCNDelay-CCA replaces DCTCP, using observed delay to set a target congestion window and ECN as a fallback so it is never more aggressive than DCTCP.Delay provides a queuing signal across switches, NIC buffers, and the host PCI bus, where ECN marking can be unavailable.Source: §4.3.1 · pp. 6–7

4.3.2 XRegion-CCA: ECN-Driven Control for Cross-Region WANs

  • XRegion-CCA replaces Cubic for cross-region traffic, adapting Swift’s additive-increase and multiplicative-decrease loop to the fraction of ECN-marked ACKs.ECN is used instead of delay because traffic-engineering reroutes can change cross-region delay independently of congestion.Source: §4.3.2 · p. 7

5 CCAaaS Orchestrator

CCAaaS extends NetEdit to deploy and update CCAs without disrupting existing connections, roll back safely, and coordinate upgrades when both endpoints must change together.

5.1 Guarantees Provided by NetEdit

  • NetEdit’s BPF pinning keeps eBPF objects running during control-plane restarts, eliminating the Cubic traffic spike seen during upgrades without pinning.The TCP iterator retroactively applies new CCA logic to active connections, assuming initialization or policy logic is idempotent.Source: §5.1 · p. 7, Figure 9 · p. 7

5.2 Introducing a Transitional CCA for Safe Rollback

  • BPF-Cubic serves as a transitional rollback CCA because ECN negotiation persists after a CCA change, while ordinary Cubic treats ECN marks as loss.Rollback loads the transitional CCA, retroactively applies it to active connections, then unregisters the faulty CCA.Source: §5.2 · pp. 7–8

5.3 Supporting CCA-Centric Orchestration

  • CCA rollouts need coordination when both endpoints must use the same CCA, so CCAaaS extends NetEdit beyond host-level deployment.This requirement applies to CCAs such as ECN-based ones that negotiate ECN during the TCP handshake.Source: §5.3 · p. 8
  • Intra-region rollouts upgrade a region together, while cross-region rollouts update region pairs and their scope resolver to apply the new CCA between upgraded regions.Source: §5.3 · p. 8

6 CCALogger

CCALogger provides fine-grained visibility into CCA state while keeping datapath work limited and predictable through RTT-aligned logging and user-space policy decisions.

6 CCALogger

  • CCALogger scales CCA observability by logging no more than once per RTT, integrating at per-RTT CCA logic, and moving selection policy to user space.These principles align logging with control timescales, use modular CCA transitions, and avoid evaluating flexible selection predicates on the datapath.Source: §6 · p. 8

6.1 Architecture

  • The control plane selects connections to monitor and retrieves their logs, while BPF-TCP data-plane functions record metadata at initialization and CCA state at RTT boundaries.A socket-cookie comparison reduces datapath selection to a single 64-bit check.Source: §6.1 · pp. 8–9

6.2 CCALogger Use Cases

  • During XRegion-CCA development, CCALogger exposed the kernel’s immediate OPEN-to-CWR transition on one ECN mark; in deployment, it confirmed over 90% of active connections used XRegion-CCA.Source: §6.2 · p. 9

7 Operational Lessons

Operational lessons cover modular CCA interfaces, fleet-scale rollout choices, eBPF lifecycle and state hazards, and kernel changes contributed by the project.

7.1 Lessons for Designing Modular CCAs

  • Modular CCA frameworks should favor specialized interfaces, expose relevant socket and cross-layer controls, and evolve existing CCAs incrementally.Generalized interfaces can force functions to recompute statistics and infer CCA properties, adding compute and storage overhead.Source: §7.1 · p. 9

7.2 Lessons for Managing CCAs at Scale

  • At scale, choose a performant default for unknown traffic, and set rollout granularity to support both problem attribution and blast-radius control.Region-pair rollout let operators attribute aggregate metric changes to XRegion-CCA; testing network features and supporting prenegotiated state also informed deployment practice.Source: §7.2 · pp. 9–10

7.3 Lessons from Evolving eBPF to Support TCP

  • Verifier approval does not prevent latent kernel bugs, and struct_ops lifecycle assumptions can fail when userspace and the kernel both manage objects.CCAaaS bound struct_ops objects to bpf_link; map versioning also addressed rollback data that was well-formed but incompatible.Source: §7.3 · p. 10

7.4 Upstream Contributions

  • Binding struct_ops objects to bpf_link reduced registration errors from about 4K hosts per week to single digits.The project also upstreamed struct_ops link semantics and a patch for live, zero-downtime connection migration between CCAs.Source: §7.4 · p. 10

8 Implementation

CCA implementations combine C and header files, with Struct_ops modularity enabling code reuse across algorithms and faster development cycles.

8 Implementation

  • Newer CCAs reuse code from earlier implementations, supporting the rapid development cycles highlighted in Table 1.For example, DCNDelay-CCA extends BPF-DCTCP with two additional hooks and approximately 340 lines of code.Source: §8 · p. 10

9 Evaluation

The evaluation measures eBPF CCA overhead, the performance and resource costs of CCAaaS-enabled algorithms, and the effects of retroactive migration and rollback. It reports production benefits alongside low overhead and operational experience at scale.

9 Evaluation

  • The evaluation spans streaming and RPC traffic, varied flow and sender counts, 50G and 100G NICs, multiple vendors and network scopes, plus 49 hosts running about 1,100 TCP connections for overhead tests.In each experiment, all hosts use the same CCA.Source: §9 · p. 10

9.1 Analysis of eBPF-based Modularity

  • Native-kernel and eBPF DCTCP and Cubic variants show negligible performance differences, though BPF-Cubic adds approximately 5 pp CPU overhead from JIT dispatch.The dispatch cost is a larger share of Cubic’s lightweight arithmetic.Source: §9.1 · p. 10, Figure 11 · p. 11

9.2 Analysis of CCAaaS-Enabled CCAs

  • DCNDelay-CCA replaced DCTCP as the default for over 94% of intra-region traffic, while XRegion-CCA replaced Cubic for 94.7% of cross-region traffic.Source: §9.2 · p. 11
  • DCNDelay-CCA adds 0.4 pp CPU over BPF-DCTCP, while XRegion-CCA reduces CPU by 2.0 pp relative to BPF-Cubic; both impose minimal fleet-wide CPU and memory overhead.Table 6 reports resource overheads at p95 and p99; memory variation across hosts reflects differences in concurrent connection counts.Source: §9.2 · p. 11, Table 6 · p. 11
  • DCNDelay-CCA improved intra-region retransmissions and packet discards, with lower CPU utilization; its smaller congestion window reduced in-network queuing and created headroom benefiting cross-region flows.Source: §9.2 · p. 11
  • XRegion-CCA reduced service latency by up to 20× for latency-sensitive cross-region workloads.For the AI-workload storage service, it significantly improved P99 latency over Cubic, especially for larger objects.Source: §9.2 · p. 11

9.3 Analysis of Retroactive Application, and Rollback

  • TCP iterator migrated over 6 million existing sockets across one region’s data centers, averaging approximately 4 ms per host with a 125–175 ms P99 and no measurable host-metric impact.Migrated connections temporarily lost 13% throughput for 20–30 ms while state variables were modified.Source: §9.3 · p. 11
  • XRegion-CCA rollback reverted five regions to BPF-Cubic one at a time, completing all regions within one day.Each region took 5–10 minutes; rollback generally takes approximately 3 minutes end-to-end.Source: §9.3 · p. 11

10 Related Work

Related work covers programmable CCA frameworks, deployment management, eBPF orchestration, and transport observability. CCAaaS focuses on managing a changing portfolio of CCAs and exposes richer CCA-internal state for diagnosis.

Modular and Programmable CCA Frameworks

  • Userspace approaches enable rapid prototyping, while CCAaaS uses eBPF for in-kernel prototyping and reuses existing CCA mechanisms; other eBPF frameworks lack its management and customization level.Source: §10 · pp. 11–12

Managing CCA Deployment at Scale

  • Prior hyperscaler reports focus on individual CCA deployments, whereas CCAaaS addresses tools and frameworks for managing a large, evolving CCA portfolio.Other evolvability work considers userspace stacks such as Snap and Luna.Source: §10 · p. 12

eBPF Management and Orchestration

  • General-purpose eBPF orchestration systems often target TC/XDP rather than struct_ops; CCAaaS extends NetEdit with CCA-specific mechanisms.Source: §10 · p. 12

Transport Observability

  • Traditional TCP tools diagnose application performance under well-behaved CCAs, while CCALogger externalizes richer CCA-internal state to diagnose incorrect behavior.Its exported data includes congestion-window, ECN, loss, and transmit-state indicators.Source: §10 · p. 12

11 Discussion

The discussion identifies boundaries for cloud use, non-struct_ops CCAs, and cross-CCA fairness, while describing possible hardware-assisted transports and BBR support requirements.

Cloud Data Centers

  • In public clouds, CCAaaS lacks control over or access to ECN signals, so the authors currently choose CCAs that do not require ECN and leave dynamic signal selection open.Source: §11 · p. 12

Programmable NICs

  • CCAaaS could complement NIC offloading, but doing so would likely require BPF control of hardware and shared memory for consistent socket or connection state.Source: §11 · p. 12

Non-struct_ops CCAs

  • NetEdit can support non-struct_ops CCAs, but consistent storage and execution benefits may be harder to realize for them.Source: §11 · p. 12

Cross-CCA Fairness

  • Ensuring fairness across CCAs is outside CCAaaS’s scope; fairness is traditionally managed through algorithm design, network configuration, or both.Source: §11 · p. 12

Supporting BBR

  • Supporting BBRv3 would require upstream BPF-TCP interface extensions because BBRv3 adds a callback, replaces another, and expands per-connection state from 104 to 144 bytes.Source: §11 · p. 12

12 Conclusion and Open Research Challenges

CCAaaS has supported five eBPF-based CCAs over five years, carrying over 97% of internal traffic. The conclusion also identifies semantic verification and interacting programs as open research challenges.

12 Conclusion and Open Research Challenges

  • Over five years, CCAaaS supported five eBPF-based CCAs carrying over 97% of Meta’s internal traffic.The authors also describe solutions tailored to their production environment and highlight open challenges for future research.Source: §12 · p. 12
  • The eBPF verifier prevents kernel crashes but does not establish program safety, especially when interoperating programs have emergent behavior.The authors call for semantic verification that considers interactions among programs, not only individual programs.Source: §12 · p. 12

P99 Latency Retransmits Thru-Flows Cubic XR-CCA Cubic XR-CCA put

  • Table 8 frames a synthetic single-receiver comparison of Cubic and XRegion-CCA using P99 latency, retransmissions, and aggregate throughput.Source: Table 8 · p. 16

B Mixed-CCA Coexistence

In a coexistence testbed, the production DCNDelay-CCA × XRegion-CCA pairing avoids cross-region starvation, while changing the intra-region CCA also changes cross-region goodput. The paper details the algorithms and identifies unresolved challenges in signal transitions, versioning, and eBPF program coexistence.

B Mixed-CCA Coexistence

  • Only the production DCNDelay-CCA × XRegion-CCA pairing avoids cross-region starvation, raising Jain’s index from 0.48 to 0.94.DCNDelay-CCA has the lowest intra-region standing queue, 15.6 KB versus 34.8 KB under Cubic. Changing only the intra-region CCA raises cross-region goodput from 0.7 to 19.9 Gbps, while intra-region goodput falls by 46% and median transaction time rises from 25.8 to 57.8 ms; aggregate throughput changes by only 6%.Source: B Mixed-CCA Coexistence · p. 16, Table 9 · p. 17
  • Each coexistence pairing was run once, cross-region RTTs reached 48–54 ms, queue occupancy was inferred from socket state, and sender kernels differed.Source: B Mixed-CCA Coexistence · p. 16

C DCNDelay-CCA Algorithm Details

  • DCNDelay-CCA combines a delay-based ideal congestion window with an ECN response that serves as a DCTCP-style guardrail.It reduces a window above target by 30% of the distance to target; absent congestion signals, it increases additively and can ramp up faster after consecutive clear periods.Source: C DCNDelay-CCA Algorithm Details · p. 17

D XRegion-CCA Algorithm Details

  • XRegion-CCA’s dynamic ECN threshold accounts for delivery rate: higher-rate flows receive a smaller effective threshold and react earlier.Its response combines CUBIC growth with DCTCP-style ECN-alpha tracking, gentler ECN-driven reductions, and its own PRR for smooth window reduction.Source: D XRegion-CCA Algorithm Details · p. 17
  • XRegion-CCA uses its own congestion state machine because a single ECN mark can trigger premature kernel rate collapse on cross-region paths.Its two states increase or reduce the congestion window; the reduction targets depend on ECN alpha or packet loss, and a timer avoids reacting too strongly to transient congestion.Source: D XRegion-CCA Algorithm Details · pp. 17–18

E Future Work

  • Sticky ECN signals complicate CCA transitions: rollback to a non-ECN CCA can make Cubic treat marked packets as implicit losses.The current migration workaround uses an intermediate CCA and fleet-wide connection iteration; the authors call for mid-connection signal teardown and draining in-flight marked packets.Source: E Future Work · p. 18
  • CCA versioning lacks explicit compatibility checks, while current workarounds rely on brittle naming and manual connection migration.The paper also describes risks from cross-program shared-map changes and proposes version metadata, compatibility declarations, and shared-state contracts.Source: E Future Work · p. 18
  • Independent eBPF programs can conflict through shared state, and cloud datacenters add heterogeneous hardware and feature support, including switches without ECN.The authors describe program chaining as an operational workaround; cloud deployment also requires feature detection, heterogeneous monitoring, and designs robust to vendor-specific behavior.Source: E Future Work · pp. 18–19
Loading 2610.33231v1…