Source-linked AI summary
Dapper: Data Plane Performance Diagnosis of TCP
Mojgan Ghasemi, Theophilus Benson, Jennifer Rexford
TL;DR
Cloud providers need timely TCP diagnosis without instrumenting tenant VMs. Dapper analyzes packets at the network edge, diagnoses sender, network, and receiver bottlenecks, and reports 94% average diagnosis accuracy.
Problem
Cloud providers need timely TCP performance diagnosis, but cannot instrument tenant VM network stacks without violating tenant control.
Source: §1 · p. 1Method
Results
94% average diagnosis accuracy was reported for Dapper, with accuracy depending on problem severity.
Source: §6.2 · p. 11Takeaways & Limitations
Dapper can help cloud providers diagnose TCP connections without instrumenting tenant VMs and can run at line rate in a hypervisor, NIC, or top-of-rack switch.
Source: §8 · p. 12RTTvar is omitted by default because adding its computation requires another pipeline stage that affects processing rate.
Source: §4.2 · p. 9
Abstract
from arXiv · showhide
With more applications moving to the cloud, cloud providers need to diagnose performance problems in a timely manner. Offline processing of logs is slow and inefficient, and instrumenting the end-host network stack would violate the tenants' rights to manage their own virtual machines (VMs). Instead, our Dapper system analyzes TCP performance in real time near the end-hosts (e.g., at the hypervisor, NIC, or top-of-rack switch). Dapper determines whether a connection is limited by the sender (e.g., a slow server competing for shared resources), the network (e.g., congestion), or the receiver (e.g., small receive buffer). Emerging edge devices now offer flexible packet processing at high speed on commodity hardware, making it possible to monitor TCP performance in the data plane, at line rate. We use P4 to prototype Dapper and evaluate our design on real and synthetic traffic. To reduce the data-plane state requirements, we perform lightweight detection for all connections, followed by heavier-weight diagnosis just for the troubled connections.
1. INTRODUCTION
Cloud providers need timely TCP performance diagnosis but cannot instrument tenant VMs, while core measurements lack an end-host perspective. Dapper monitors TCP at the network edge and balances diagnostic detail against data-plane resource limits.
1. INTRODUCTION
- Cloud providers need timely, accurate performance visibility to balance infrastructure utilization with tenant performance, but cannot modify tenant VM network stacks.Core measurements also lack end-host perspective and may require combining data from multiple switches.Source: §1 · p. 1
- Programmable edge devices make line-rate data-plane diagnosis possible, but metrics must balance diagnostic detail against limited device capabilities.Dapper must also handle heterogeneous TCP versions and tuned parameters.Source: §1 · p. 1
- Dapper analyzes TCP packets in real time at the network edge to quantify sender, network, and receiver contributions to poor performance.Identifying the responsible entity can take hours to days; specialized tools can then pinpoint the root cause.Source: §1 · p. 1
- Dapper reduces data-plane resource demands by lowering the accuracy of some measurements and collecting lightweight metrics for all flows before heavier monitoring of troubled flows.The data plane can also use the inferred information for quick decisions such as load balancing network-limited connections.Source: §1 · p. 2
2. TCP PERFORMANCE MONITORING
Dapper monitors server-side TCP traffic and infers sender, network, and receiver metrics from packets to diagnose bottlenecks. Its methods estimate application sending behavior, congestion windows, and receiver constraints without requiring tenant cooperation.
2. TCP PERFORMANCE MONITORING
- Monitoring near the server exposes both directions of a connection and supports imitation of the server’s congestion-control state and response timing.A connection is represented by the bidirectional four-tuple of server and client IP addresses and port numbers.Source: §2 · p. 2
2.1 Inferring Sender Statistics
- Sender limitation is assessed by comparing sent packets with potential sending rate, while segment sizes and ACK-to-data reaction time indicate whether the application is backlogged.Reaction time is measured between a new ACK’s arrival and transmission of new data, using an empirically derived threshold based on edge-to-VM latency.Source: §2.1 · pp. 2–3
2.2 Inferring Network Statistics
- Dapper classifies retransmissions as fast retransmit or timeout using duplicate ACKs and estimates latency from packet timestamps and acknowledgments.It uses Karn’s algorithm for smoothed RTT, RTT variation, and retransmission timeout estimates; retransmitted packets are excluded from RTT sampling.Source: §2.2 · pp. 3–4
- Because TCP variants and changing thresholds complicate congestion-window estimation, Dapper tracks flight size and adjusts its lower-bound inf_cwnd estimate according to loss type.The estimate rises with new flight-size maxima, decreases multiplicatively after fast recovery, and resets to the initial window after a timeout.Source: §2.2 · p. 4
2.3 Inferring Receiver Statistics
- Receiver limitation is measured by tracking advertised receive windows and window scaling, while multiple tuples acknowledged at once indicate delayed ACK behavior.The packet-processing logic hashes the four-tuple, extracts direction-specific fields, and updates relevant per-flow metrics.Source: §2.3 · p. 4
3. TCP DIAGNOSIS TECHNIQUES
Dapper diagnoses sender-, network-, and receiver-side limits using TCP behavior, including sending patterns, window utilization, RTT, loss, and acknowledgment timing. A loss-recovery heuristic adds evidence about sender backlog under specified assumptions.
Diagnosing Sender Problems
- Dapper identifies a non-backlogged sender when it sends less than the allowed window, transmits sub-MSS packets, or has reaction times above an empirical threshold.The diagnosis also considers sending-rate growth and flight-size changes during loss recovery.Source: §3 · p. 5
- During loss recovery, flight-size change can indicate sender backlog, assuming the connection is not receiver-limited and the sender’s data-generation rate remains unchanged.The examples also assume the sender’s state remains unchanged during recovery.Source: §3 · p. 5
Diagnosing Network Problems
- A network bottleneck is indicated when loss lowers the congestion-window estimate below the receive window; RTT is compared with an expected or minimum per-flow baseline for latency problems.The section also frames network limits in terms of bandwidth, packet loss, and increased latency.Source: §3 · pp. 5–6
Diagnosing Receiver Problems
- A receiver is diagnosed as limiting performance when its receive window is below the congestion-window estimate, or when each ACK frees more than one RTT sample on average.These conditions correspond to a small receive window and delayed acknowledgments, respectively.Source: §3 · p. 6
4. DATA-PLANE MONITORING
Dapper uses P4 to monitor TCP connections in the data plane, with packet parsing, metadata, tables, and registers supporting per-flow tracking. Its target-independent prototype can run where an edge element supports P4.
4. DATA-PLANE MONITORING
- The prototype can run in a public cloud on varied targets if at least one edge element—the switch, hypervisor, or NIC—can run P4.The design addresses target-specific resource constraints while presenting a target-independent solution.Source: §4.1 · p. 6, §4 · p. 6
4.1 TCP Monitoring Prototype in P4
- Dapper uses flexible parsing, metadata to carry packet information across processing stages, and registers to retain per-flow state across packets.P4 tables and actions perform packet operations, while flow control tests conditions and selects tables.Source: §4.1 · pp. 6–7
- The prototype assumes Ethernet, IPv4, and TCP headers, while the authors say it can be extended to other protocols such as IPv6.It also parses TCP options, including MSS, using a fixed de-parsing order.Source: §4.1 · p. 6
- A custom symmetric hash maps both directions of a connection to the same flow index, enabling cross-packet analysis without target-dependent symmetric-hash support.The hash orders address and port fields consistently using a comparison of the source and destination IPs.Source: §4.1 · p. 7
5. Realizing operations using actions and tables: To re-
Dapper realizes packet monitoring with P4 tables, actions, and control flow, using metadata to bridge conditional logic across pipeline stages. Hardware constraints lead to compromises in collision handling, RTT sampling, register access, window scaling, and RTT variability.
5. Realizing operations using actions and tables: To re-
- The packet-processing logic hashes packets to a flow index, initializes flows on their first packet, and branches on packet direction to update monitoring state.The code includes branches for ACKs, RTT sampling, retransmissions, and forwarding.Source: §5 · pp. 7–8
- P4 tables and actions implement packet-processing operations, while control flow applies tables and tests conditions to select the relevant processing path.Because conditions are restricted to control segments, a loader table places register values in metadata for later tests.Source: §5 · pp. 7–8
4.2 Hardware Resource Constraints
- A 32-bit hash is used, but hardware omits stored connection tuples; sequence-number and window checks provide a sanity flag for potentially compromised statistics.The flag can be queried alongside connection metrics to indicate whether diagnosis statistics are reliable.Source: §4.2 · p. 8
- To limit per-flow state, the P4 program keeps at most one outstanding RTT tuple and samples an outgoing packet only when its queue is empty.The software design otherwise maintains a queue of sequence-number and timestamp tuples for RTT measurements.Source: §4.2 · p. 8
- Register accesses from multiple tables limit processing rate; the authors note that a future P4 ternary operator could enable conditional assignments without global registers.The processing-rate constraint arises because some registers are accessed in multiple tables.Source: §4.2 · p. 8
- Because most P4 targets shift only by constants, the prototype records both advertised RWND and its scale value for the control plane to combine.Source: §4.2 · pp. 8–9
- RTTvar is omitted by default because computing it requires an unsupported absolute-value operation or an added pipeline stage that impacts processing rate.The stage can be enabled optionally.Source: §4.2 · p. 9
5. TWO-PHASE TCP MONITORING
Dapper reduces monitoring cost by tracking lightweight metrics for every connection and enabling heavier diagnosis only after a flow is flagged as troubled. Midstream inference can further reduce state, but trades some measurement accuracy for that savings.
5. TWO-PHASE TCP MONITORING
- Two-phase monitoring keeps lightweight detection active for all connections and enables heavyweight diagnosis only when a connection meets the badness criterion.The first phase uses average flow rate as a general indicator of performance; the second collects fuller TCP statistics to identify the limiting component.Source: §5 · p. 9
- Average flow rate is the detection metric, maintained with registers for start time, bytes sent, and last update time.The control plane can compare per-flow state against a tenant-specific threshold; low rate or a tenant complaint can trigger heavyweight monitoring.Source: §5 · p. 9
- Unknown midstream packet counters can make inferred flight size inaccurate, affecting estimates of inf_cwnd.The method tracks outgoing sequence numbers and incoming ACKs to infer flight size and inf_cwnd.Source: §5 · p. 9
- Midstream inference reduces memory overhead but sacrifices accuracy; retaining handshake TCP options for all flows is more accurate and requires more data-plane state.Midstream MSS inference tracks the largest observed segments, while window-scale inference uses flight size and unscaled receive-window advertisements.Source: §5 · p. 9
6. EVALUATION
The evaluation measures Dapper’s memory and CPU overhead, diagnosis accuracy on synthetic traffic, and findings on CAIDA traces. It also examines how memory-saving measurement choices affect accuracy.
6.1 CPU and Memory Overhead
- For 10K connections, single-phase monitoring requires 670 KB of state, while two-phase monitoring uses about 16 bytes per flow in its first phase.Figure 7 compares switch state for single-phase monitoring with two-phase monitoring when 10% or 20% of connections are troubled.Source: §6.1 · pp. 9–10, Figure 7 · p. 10
- A table of at least 262,144 entries tracks 10K connections with a collision rate below 4%, using less than 18 MB on the switch.The expected collision rate rises with the number of flows; Figure 8 plots it for different flow counts and table sizes.Source: Figure 8 · p. 10, §6.1 · p. 10
- Dapper’s CPU consumption is close to approaches using near-real-time polling at 50 ms frequency as aggregate bandwidth increases.Figure 9 plots CPU consumption against aggregate bandwidth processed.Source: Figure 9 · p. 10, §6.1 · p. 10
- The first packet usually takes longer to process because Dapper allocates and initializes flow state and parses packet options.Collisions, retransmissions, duplicate ACKs, and packets used for RTT measurement also add processing work.Source: Figure 10 · p. 10, §6.1 · p. 10
6.2 Diagnosis Accuracy
- 94% average diagnosis accuracy; accuracy rises with problem severity because problem detection depends on severity.Figure 11 illustrates the severity relationship; Table 2 reports diagnosis sensitivity and accuracy.Source: Table 2 · p. 11, Figure 11 · p. 11, §6.2 · p. 11
6.3 Analyzing CAIDA Traces
- About 90% of CAIDA flows spend some time network-limited, and nearly half of those are network-limited 50% of the time.The analysis considers 244,185 flows and disables application-reaction inference because these traces were collected in the network core; the authors therefore expect the actual sender-limited rate to be higher.Source: Figure 12 · p. 11, §6.3 · p. 11
6.4 Trade-offs in Accuracy and Overhead
- With a queue size of one, Dapper needs more samples to reduce SRTT error; an unbounded queue increases memory use by about 9%.The traces are from a WAN, not a data center.Source: §6.4 · pp. 11–12, Figure 13 · p. 12
- Midstream window-scale inference differs from ground truth by about 20% when flows are generally not receive-window-limited.In those flows, observed flight sizes do not approach the receiver-imposed upper bound used to estimate the scale.Source: §6.4 · p. 12
7. RELATED WORK
Dapper is contrasted with offline trace analysis, coarse core measurement, end-host instrumentation, tomography, and ML-based diagnosis. The paper emphasizes edge visibility and TCP-derived techniques.
7. RELATED WORK
- Offline trace tools target known TCP variants or store headers for later queries, but are unsuitable for real-time diagnosis and add collection overhead.Source: §7 · p. 12
- Core-switch measurements can be too coarse for diagnosing poor performance, while Dapper relies on edge visibility for a simpler solution.The related-work discussion also contrasts this with tools coordinating switches to analyze the full packet stream.Source: §7 · p. 12
- End-host techniques suit private clouds but are too invasive for public IaaS; Dapper infers VM state at the edge without tenant cooperation.Source: §7 · p. 12
- Unlike tomography’s inference of link properties from end-to-end measurements, Dapper measures performance directly and continuously; tomography can help locate congested links.Source: §7 · p. 12
- Dapper derives techniques from TCP characteristics and is agnostic to applications and TCP variants, whereas ML approaches require training for each variant.Source: §7 · p. 12
- Related edge work virtualizes congestion control in the hypervisor or enforces it in the vswitch without tenant cooperation.Source: §7 · p. 12
8. CONCLUSION
Dapper diagnoses TCP connections without instrumenting tenant VMs. It can run at line rate in the hypervisor, NIC, or top-of-rack switch.
8. CONCLUSION
- Dapper helps cloud providers diagnose TCP connections.Source: §8 · p. 12
- Dapper performs this diagnosis without instrumenting tenant VMs.Source: §8 · p. 12
- Dapper can run at line rate in the hypervisor, NIC, or top-of-rack switch.Source: §8 · p. 12