Source-linked AI summary
Operation-Type-Aware Client Routing for Leader-Based Consensus Datastores
Sri Saran Balaji Vellore Rajakumar, James Thompson
TL;DR
Leader-based consensus datastores need routing that respects the different protocol roles of writes and linearizable reads, but client latency blends their costs. The paper implements operation-aware routing that pins writes to the leader and distributes reads across healthy members, then evaluates it against standard policies in etcd and ZooKeeper. It reports lower write latency and higher throughput in steady state, with larger gains when a follower degrades, while finding that latency remains useful mainly for removing bad read endpoints.
Problem
Client routing treats consensus members too uniformly, while blended latency does not reveal the distinct costs of follower-forwarded writes and leader-confirmed local reads.
Method
The paper compares round_robin, latency-adaptive routing, and hybrid routing that pins writes to the leader and distributes reads across healthy members.
Results
Hybrid lowers write P50 by 29.4% and raises throughput by 8.8% on the 80/20 3-node workload, while degraded-follower trials cut read P99 by 64.2% and write P99 by 73.9%.
Takeaways & Limitations
Routing by operation semantics preserves distributed healthy reads, removes unnecessary write forwarding, and uses latency narrowly to exclude clearly degraded read endpoints.
Takeaways & Limitations
Latency-only routing cannot separate read-serving cost from write-forwarding cost, and the steady-state measurements assume a fixed leader; leadership changes still require active refresh and fallback.
Abstract
from arXiv · showhide
Leader-based consensus datastores (etcd, ZooKeeper) face two competing routing goals: spread load evenly across members, and route operations to the member whose protocol role matches the operation. Writes must commit through the leader, so sending them elsewhere adds a forwarding hop. Linearizable reads need only a lightweight leader confirmation before any member can serve them locally. The upstream etcd client uses gRPC's round_robin balancer, distributing reads and writes uniformly across cluster members. An operation-aware client resolves this by pinning writes to the leader and distributing reads across the healthy read pool. In steady state on a 3-node etcd cluster (80/20 read/write mix, 5 trials), this lowers write P50 by 29% and raises throughput by 9%. When a follower degrades silently, the operation-aware client detects the latency shift and removes it from the read pool, cutting read P99 by 64%, write P99 by 74%, and raising throughput by 89%. The same routing rule applied to ZooKeeper (ZAB protocol, different implementation) points in the same direction, showing that the result follows from leader-based consensus structure rather than one system's implementation. The key obstacle to discovering this policy adaptively is that the leader confirmation round-trip occurs between cluster members, so the client sees only a blended latency signal rather than the decisive coordination cost directly.
I. INTRODUCTION
Leader-based datastores must balance healthy read distribution against leader-required writes. The paper argues for operation-aware routing because follower-forwarded writes add cost, while followers can serve linearizable reads after leader confirmation.
- Motivation: The stock client is only partly correct: it distributes healthy reads but sends writes uniformly and continues using degraded read targets.This behavior follows from treating members as interchangeable despite their protocol roles.
- Operation-aware routing: The proposed routing policy pins writes to the leader while distributing reads across healthy members.A read-side monitor removes a member only when it becomes a clear latency outlier.
- Operation-type asymmetry: Writes commit through the leader, so routing them to followers adds a forwarding hop without benefit.On three nodes, round_robin sends 2/3 of writes through this extra path.
- Operation-type asymmetry: Linearizable reads require leader confirmation but can be served from local follower state.The serving member waits until its applied state reaches the confirmed commit index.
B. Consensus-Cost Invisibility
Client-observed latency blends read-serving and write-forwarding costs, so latency adaptation alone cannot recover the correct steady-state routing split. The paper instead separates operation semantics from latency-based degradation detection.
- Consensus-cost invisibility: A nearby follower can look fastest because many fast reads dominate the blended signal while fewer writes still incur hidden forwarding.The hidden coordination cost is paid inside the cluster rather than cleanly exposed to the client.
- Adaptive role of latency: Latency remains useful for detecting degraded members that should leave the read pool.Its role is narrower than selecting the steady-state destination for mixed traffic.
- Routing policies: The proposed comparison tests round_robin, latency-adaptive boltzmann, and operation-aware hybrid under identical conditions.Hybrid sends writes to the leader and distributes reads across healthy members.
- Operation asymmetry: Leader-based consensus gives writes and linearizable reads opposite routing requirements.Writes require leader commitment, whereas reads can be served locally after confirmation.
B. Linearizable Reads Need Leader Confirmation but Followers Serve
Linearizable reads coordinate with the leader without requiring the leader to serve the query. This permits distributed local serving in etcd and ZooKeeper, while preserving the operation-aware distinction from writes.
- etcd ReadIndex: In etcd, ReadIndex confirms a commit index with the leader before the follower serves the read locally.The follower waits until its applied state reaches the confirmed index.
- ZooKeeper ZAB: ZooKeeper’s sync() provides the analogous coordination step before a follower serves a linearizable read.The connected follower catches up to the leader’s latest committed transaction, then reads from its own copy.
- Cross-system implication: The shared mechanism supports distributing linearizable reads across members while routing writes to the leader.The routing rule therefore extends across the two consensus implementations.
- Existing clients: General-purpose clients and proxies treat members as interchangeable rather than adapting destinations to consensus roles.The etcd client uses round_robin, while ZooKeeper’s client stays with one selected server until reconnect.
III. ROUTING POLICIES
The evaluation compares a uniform baseline, a blended-latency adaptive policy, and a role-aware policy. Hybrid separates endpoint selection by operation type while retaining latency-based removal of degraded read targets.
- Hybrid: Hybrid sends writes to the leader and distributes reads across the healthy read pool.This is the proposed operation-aware policy.
- boltzmann: boltzmann selects endpoints using observed latency for both reads and writes.It serves as the natural adaptive baseline and uses one blended signal.
- Benchmark procedure: Each request is assigned a read or write type, routed to an endpoint, executed through the etcd client, and recorded for latency feedback.All policies use equal connection counts for fair comparison.
- round_robin: round_robin uniformly distributes every operation across endpoints.It applies the same destination rule to reads and writes.
C. Boltzmann Exploration
Boltzmann adapts endpoint selection from recent latency, but mixed read/write traffic can make a fast follower appear preferable even when its writes incur forwarding. Hybrid instead separates operation types, sending writes to the leader and distributing reads across healthy members while retaining latency-based outlier removal.
- Boltzmann Exploration: Boltzmann converts EWMA latency estimates into endpoint-selection probabilities, using softmax temperature to balance exploitation and exploration.Errors inflate estimates, and endpoints with more than three consecutive failures receive near-zero probability.
- Boltzmann Exploration: In mixed workloads, fast follower reads can dominate the latency signal, causing Boltzmann to continue favoring a follower whose writes take the wrong path.The policy lacks an operation-type distinction, so fewer forwarded writes may not shift the average enough.
- Hybrid Routing: Hybrid routes writes to the detected leader, eliminating the forwarding hop.Leader detection uses a cached leader refreshed by periodic Status() polling.
- Hybrid Routing: Hybrid distributes reads across the healthy read pool and removes a member when its read EWMA exceeds the cluster-median EWMA by the configured threshold.When all endpoints are healthy, the policy reduces to leader-pinned writes and distributed reads; degradation triggers read-pool cleanup.
A. Infrastructure
The evaluation uses repeated measurements on configured 3-node and 5-node etcd clusters, with stable-leader steady-state comparisons and a default 80/20 linearizable read/write workload. Hybrid improves write performance over round_robin while preserving distributed reads, with larger gains in the more write-heavy rerun.
- Infrastructure: The main experiments use 3-node and 5-node etcd clusters with matched instance and storage configurations, while the client remains in the same region.The 3-node cluster spans multiple availability zones; the 5-node scale-up uses the same per-node configuration.
- Workload: Experiments normally run for 60 seconds with 128 workers using 80% linearizable reads and 20% single-key writes over a 1,000-key space.Connection budgets are held equal across routing policies.
- Measurement Scope: The primary quantitative comparisons use repeated trials, while the leader remains unchanged during the measured steady-state intervals.Elections and representative mechanism sweeps are treated separately from the repeated-trial comparisons.
- Main Result: 29.4%: hybrid lowers write P50 from 6.22 ms to 4.39 ms on the 80/20 workload while read P50 remains close to round_robin.Throughput rises from 14,536 to 15,809 ops/s, an 8.8% gain.
- Main Result: 31.3%: hybrid lowers write P50 from 8.57 ms to 5.89 ms in the 50/50 rerun and raises throughput by 13.7%.The more write-heavy workload makes the operation-type split more valuable.
- Comparators: Latency-adaptive Boltzmann is used as a diagnostic comparator because adaptation helps with visibly bad endpoints but does not replace operation-aware write routing.The principal comparison is round_robin versus hybrid.
E. Read-Size Sweep Under Pure Reads
Under pure reads, distributed linearizable reads outperform leader-only reads across response sizes, with the advantage widening for larger responses. The surrounding experiments also connect this result to healthy read distribution and to removing degraded followers from the read pool.
- Read-Size Sweep: The performance gap widens as larger responses stress the leader’s local storage path.The figure compares distributed linearizable reads with leader-only reads across response sizes.
- Routing Mechanism: Distributed reads improve read serving by avoiding concentration on the leader, complementing the write benefit from eliminating forwarding.The two effects separate the read and write halves of the routing rule.
- Degraded Follower: 8 ms delay with 2 ms jitter: the degraded-follower study evaluates read-pool cleanup while the leader remains unchanged.The experiment compares round_robin and hybrid over 5 trials.
- Degraded Follower: 64.2%: hybrid cuts read P99 while nearly eliminating reads to the slow follower and preserving leader-pinned writes.The same result cuts write P99 by 73.9% and raises throughput by 89.1%.
H. Scaling to a 5-Node, 3-AZ Cluster
At five nodes across three availability zones, hybrid routing preserves the operation-aware advantage: writes improve over round_robin while distributed reads remain close to stock. The write-latency gap widens as concurrency rises, indicating that follower-forwarded writes remain costly under load.
- 36.7% lower write P50 and 8.0% higher throughput make hybrid routing beneficial in the 5-node, 80/20 workload.Across 5 trials, write P50 falls from 6.67 ms to 4.22 ms.
- 4/5 of round_robin writes land on followers at five nodes, compared with 2/3 at three nodes, so scale does not remove forwarding cost.
- The 128-to-512-worker sweep shows round_robin write P50 rising from 6.72 ms to 9.70 ms, versus 4.22 ms to 6.12 ms for hybrid.
- The absolute write-latency gap grows from 2.50 ms to 3.58 ms as concurrency increases.
J. A Second System in Apache ZooKeeper
The ZooKeeper evaluation reproduces the operation-aware routing pattern in a different leader-based consensus implementation. Hybrid has the lowest write latency and remains substantially better than a sticky follower session, especially when that follower degrades.
- 6.9% lower write P50 gives hybrid the lowest write latency in the 3-node ZooKeeper experiment.Hybrid records 3.67 ms versus 3.94 ms for round_robin across 3 trials.
- Hybrid lowers healthy sticky-session read P50 from 12.10 ms to 7.13 ms and write P50 from 6.32 ms to 3.97 ms.
- Under an 8 ms delay and 2 ms jitter on the pinned follower, hybrid lowers read P50 by 82.6% and write P50 by 80.7%.
- Throughput rises from 3.08k to 16.91k ops/s under the degraded-follower condition, a 5.5× improvement for hybrid over the sticky session.
- The kube-apiserver experiment is supporting evidence rather than the headline result because apiserver-side work and uncached read handling dilute read-heavy gains.
V. DISCUSSION
The discussion frames routing as a consequence of operation-specific consensus work: distribute linearizable reads, direct writes to the leader, and remove degraded members from the read pool. It also limits the claim to the evaluated settings and distinguishes routing from orthogonal caching and topology mechanisms.
- ReadIndex makes distributed linearizable reads appropriate because followers confirm freshness and serve responses locally, while writes incur forwarding when sent to followers.
- Latency is useful for detecting degradation but cannot reliably select the correct steady-state endpoint for mixed linearizable traffic.
- Hybrid reduces a delayed follower’s read share from one-third to 0.5%, improving both read and write tails.
- The evaluated scope centers on 3-node and 5-node etcd clusters, with supporting ZooKeeper checks rather than a claimed full scaling law.
- A follow-on etcd design proposes advisory leader hints and an opt-in balancer that directs leader-dependent operations while retaining fallback behavior.
- Caching and topology-aware mechanisms are complementary because they reduce requests or cross-zone traffic but do not separate reads from writes.
VII. CONCLUSION
The conclusion recommends routing according to the protocol work induced by each operation: direct writes to the leader, distribute reads across healthy members, and use latency to detect degradation rather than infer steady-state routing. The reported gains are strongest for degraded followers, while the steady-state measurements assume a fixed leader.
- Latency-only routing fails to separate read-serving cost from write-forwarding cost, although it remains useful for detecting degradation.
- 64.2% lower read P99, 73.9% lower write P99, and 89.1% higher throughput are achieved under a degraded same-AZ follower.
- 29.4% lower write P50 on 3 nodes and 36.7% on 5 nodes accompany distributed linearizable reads in healthy steady-state measurements.