PaperlayerView exact revision on arXiv

Source-linked AI summary

Operation-Type-Aware Client Routing for Leader-Based Consensus Datastores

Sri Saran Balaji Vellore Rajakumar, James Thompson

arXiv:2609.00392v1cs.DCcs.SE

TL;DR

Leader-based consensus datastores need routing that respects the different protocol roles of writes and linearizable reads, but client latency blends their costs. The paper implements operation-aware routing that pins writes to the leader and distributes reads across healthy members, then evaluates it against standard policies in etcd and ZooKeeper. It reports lower write latency and higher throughput in steady state, with larger gains when a follower degrades, while finding that latency remains useful mainly for removing bad read endpoints.

Abstract

from arXiv · show

Leader-based consensus datastores (etcd, ZooKeeper) face two competing routing goals: spread load evenly across members, and route operations to the member whose protocol role matches the operation. Writes must commit through the leader, so sending them elsewhere adds a forwarding hop. Linearizable reads need only a lightweight leader confirmation before any member can serve them locally. The upstream etcd client uses gRPC's round_robin balancer, distributing reads and writes uniformly across cluster members. An operation-aware client resolves this by pinning writes to the leader and distributing reads across the healthy read pool. In steady state on a 3-node etcd cluster (80/20 read/write mix, 5 trials), this lowers write P50 by 29% and raises throughput by 9%. When a follower degrades silently, the operation-aware client detects the latency shift and removes it from the read pool, cutting read P99 by 64%, write P99 by 74%, and raising throughput by 89%. The same routing rule applied to ZooKeeper (ZAB protocol, different implementation) points in the same direction, showing that the result follows from leader-based consensus structure rather than one system's implementation. The key obstacle to discovering this policy adaptively is that the leader confirmation round-trip occurs between cluster members, so the client sees only a blended latency signal rather than the decisive coordination cost directly.

I. INTRODUCTION

Leader-based datastores must balance healthy read distribution against leader-required writes. The paper argues for operation-aware routing because follower-forwarded writes add cost, while followers can serve linearizable reads after leader confirmation.

Motivation

  • The stock client is only partly correct: it distributes healthy reads but sends writes uniformly and continues using degraded read targets.This behavior follows from treating members as interchangeable despite their protocol roles.Source: §I · p. 1, A. The Operation-Type Asymmetry · p. 2

Operation-aware routing

Operation-type asymmetry

  • Writes commit through the leader, so routing them to followers adds a forwarding hop without benefit.On three nodes, round_robin sends 2/3 of writes through this extra path.Source: A. The Operation-Type Asymmetry · p. 1
  • Linearizable reads require leader confirmation but can be served from local follower state.The serving member waits until its applied state reaches the confirmed commit index.Source: A. The Operation-Type Asymmetry · pp. 1–2

B. Consensus-Cost Invisibility

Client-observed latency blends read-serving and write-forwarding costs, so latency adaptation alone cannot recover the correct steady-state routing split. The paper instead separates operation semantics from latency-based degradation detection.

Consensus-cost invisibility

  • A nearby follower can look fastest because many fast reads dominate the blended signal while fewer writes still incur hidden forwarding.The hidden coordination cost is paid inside the cluster rather than cleanly exposed to the client.Source: B. Consensus-Cost Invisibility · p. 2

Adaptive role of latency

  • Latency remains useful for detecting degraded members that should leave the read pool.Its role is narrower than selecting the steady-state destination for mixed traffic.Source: B. Consensus-Cost Invisibility · p. 2

Routing policies

  • The proposed comparison tests round_robin, latency-adaptive boltzmann, and operation-aware hybrid under identical conditions.Hybrid sends writes to the leader and distributes reads across healthy members.Source: C. Contributions · p. 2

Operation asymmetry

B. Linearizable Reads Need Leader Confirmation but Followers Serve

Linearizable reads coordinate with the leader without requiring the leader to serve the query. This permits distributed local serving in etcd and ZooKeeper, while preserving the operation-aware distinction from writes.

etcd ReadIndex

  • In etcd, ReadIndex confirms a commit index with the leader before the follower serves the read locally.The follower waits until its applied state reaches the confirmed index.Source: B. Linearizable Reads Need Leader… · p. 3

ZooKeeper ZAB

  • ZooKeeper’s sync() provides the analogous coordination step before a follower serves a linearizable read.The connected follower catches up to the leader’s latest committed transaction, then reads from its own copy.Source: C. The Same Mechanism in ZooKeeper (ZAB) · p. 3

Cross-system implication

  • The shared mechanism supports distributing linearizable reads across members while routing writes to the leader.The routing rule therefore extends across the two consensus implementations.Source: C. The Same Mechanism in ZooKeeper (ZAB) · p. 3

Existing clients

  • General-purpose clients and proxies treat members as interchangeable rather than adapting destinations to consensus roles.The etcd client uses round_robin, while ZooKeeper’s client stays with one selected server until reconnect.Source: D. Client Load Balancing · p. 3

III. ROUTING POLICIES

The evaluation compares a uniform baseline, a blended-latency adaptive policy, and a role-aware policy. Hybrid separates endpoint selection by operation type while retaining latency-based removal of degraded read targets.

Hybrid

  • Hybrid sends writes to the leader and distributes reads across the healthy read pool.This is the proposed operation-aware policy.Source: §III · p. 4

boltzmann

  • boltzmann selects endpoints using observed latency for both reads and writes.It serves as the natural adaptive baseline and uses one blended signal.Source: §III · p. 4

Benchmark procedure

  • Each request is assigned a read or write type, routed to an endpoint, executed through the etcd client, and recorded for latency feedback.All policies use equal connection counts for fair comparison.Source: A. Request Data Path · p. 4

round_robin

  • round_robin uniformly distributes every operation across endpoints.It applies the same destination rule to reads and writes.Source: B. Round Robin (Baseline) · p. 4

C. Boltzmann Exploration

Boltzmann adapts endpoint selection from recent latency, but mixed read/write traffic can make a fast follower appear preferable even when its writes incur forwarding. Hybrid instead separates operation types, sending writes to the leader and distributing reads across healthy members while retaining latency-based outlier removal.

Boltzmann Exploration

  • Boltzmann converts EWMA latency estimates into endpoint-selection probabilities, using softmax temperature to balance exploitation and exploration.Errors inflate estimates, and endpoints with more than three consecutive failures receive near-zero probability.Source: C. Boltzmann Exploration · p. 4
  • In mixed workloads, fast follower reads can dominate the latency signal, causing Boltzmann to continue favoring a follower whose writes take the wrong path.The policy lacks an operation-type distinction, so fewer forwarded writes may not shift the average enough.Source: C. Boltzmann Exploration · p. 4

Hybrid Routing

  • Hybrid routes writes to the detected leader, eliminating the forwarding hop.Leader detection uses a cached leader refreshed by periodic Status() polling.Source: D. Hybrid Routing Sends Writes to the… · pp. 4–5
  • Hybrid distributes reads across the healthy read pool and removes a member when its read EWMA exceeds the cluster-median EWMA by the configured threshold.When all endpoints are healthy, the policy reduces to leader-pinned writes and distributed reads; degradation triggers read-pool cleanup.Source: D. Hybrid Routing Sends Writes to the… · pp. 4–5

A. Infrastructure

The evaluation uses repeated measurements on configured 3-node and 5-node etcd clusters, with stable-leader steady-state comparisons and a default 80/20 linearizable read/write workload. Hybrid improves write performance over round_robin while preserving distributed reads, with larger gains in the more write-heavy rerun.

Infrastructure

  • The main experiments use 3-node and 5-node etcd clusters with matched instance and storage configurations, while the client remains in the same region.The 3-node cluster spans multiple availability zones; the 5-node scale-up uses the same per-node configuration.Source: A. Infrastructure · p. 5

Workload

  • Experiments normally run for 60 seconds with 128 workers using 80% linearizable reads and 20% single-key writes over a 1,000-key space.Connection budgets are held equal across routing policies.Source: B. Workload · p. 5

Measurement Scope

  • The primary quantitative comparisons use repeated trials, while the leader remains unchanged during the measured steady-state intervals.Elections and representative mechanism sweeps are treated separately from the repeated-trial comparisons.Source: B. Workload · p. 5

Main Result

  • 29.4%: hybrid lowers write P50 from 6.22 ms to 4.39 ms on the 80/20 workload while read P50 remains close to round_robin.Throughput rises from 14,536 to 15,809 ops/s, an 8.8% gain.Source: C. Main In-Region Result · p. 5
  • 31.3%: hybrid lowers write P50 from 8.57 ms to 5.89 ms in the 50/50 rerun and raises throughput by 13.7%.The more write-heavy workload makes the operation-type split more valuable.Source: C. Main In-Region Result · p. 5

Comparators

  • Latency-adaptive Boltzmann is used as a diagnostic comparator because adaptation helps with visibly bad endpoints but does not replace operation-aware write routing.The principal comparison is round_robin versus hybrid.Source: D. Boltzmann as a Supporting Comparator · p. 5

E. Read-Size Sweep Under Pure Reads

Under pure reads, distributed linearizable reads outperform leader-only reads across response sizes, with the advantage widening for larger responses. The surrounding experiments also connect this result to healthy read distribution and to removing degraded followers from the read pool.

Read-Size Sweep

Routing Mechanism

  • Distributed reads improve read serving by avoiding concentration on the leader, complementing the write benefit from eliminating forwarding.The two effects separate the read and write halves of the routing rule.Source: E. Read-Size Sweep Under Pure Reads · p. 6

Degraded Follower

  • 8 ms delay with 2 ms jitter: the degraded-follower study evaluates read-pool cleanup while the leader remains unchanged.The experiment compares round_robin and hybrid over 5 trials.Source: G. Degraded Same-AZ Follower · p. 6
  • 64.2%: hybrid cuts read P99 while nearly eliminating reads to the slow follower and preserving leader-pinned writes.The same result cuts write P99 by 73.9% and raises throughput by 89.1%.Source: G. Degraded Same-AZ Follower · p. 6

H. Scaling to a 5-Node, 3-AZ Cluster

At five nodes across three availability zones, hybrid routing preserves the operation-aware advantage: writes improve over round_robin while distributed reads remain close to stock. The write-latency gap widens as concurrency rises, indicating that follower-forwarded writes remain costly under load.

  • 36.7% lower write P50 and 8.0% higher throughput make hybrid routing beneficial in the 5-node, 80/20 workload.Across 5 trials, write P50 falls from 6.67 ms to 4.22 ms.Source: H. Scaling to a 5-Node, 3-AZ Cluster · p. 6
  • 4/5 of round_robin writes land on followers at five nodes, compared with 2/3 at three nodes, so scale does not remove forwarding cost.Source: H. Scaling to a 5-Node, 3-AZ Cluster · pp. 6–7
  • The 128-to-512-worker sweep shows round_robin write P50 rising from 6.72 ms to 9.70 ms, versus 4.22 ms to 6.12 ms for hybrid.Source: §I · p. 7
  • The absolute write-latency gap grows from 2.50 ms to 3.58 ms as concurrency increases.Source: §I · p. 7, Figure 6 · p. 7

J. A Second System in Apache ZooKeeper

The ZooKeeper evaluation reproduces the operation-aware routing pattern in a different leader-based consensus implementation. Hybrid has the lowest write latency and remains substantially better than a sticky follower session, especially when that follower degrades.

V. DISCUSSION

The discussion frames routing as a consequence of operation-specific consensus work: distribute linearizable reads, direct writes to the leader, and remove degraded members from the read pool. It also limits the claim to the evaluated settings and distinguishes routing from orthogonal caching and topology mechanisms.

  • ReadIndex makes distributed linearizable reads appropriate because followers confirm freshness and serve responses locally, while writes incur forwarding when sent to followers.Source: A. Why the Split Works · p. 8
  • Latency is useful for detecting degradation but cannot reliably select the correct steady-state endpoint for mixed linearizable traffic.Source: B. What the Latency Signal Should Do · p. 8, §VI · p. 9
  • Hybrid reduces a delayed follower’s read share from one-third to 0.5%, improving both read and write tails.Source: B. What the Latency Signal Should Do · p. 8
  • The evaluated scope centers on 3-node and 5-node etcd clusters, with supporting ZooKeeper checks rather than a claimed full scaling law.Source: C. Scope · p. 8, §VI · p. 8
  • A follow-on etcd design proposes advisory leader hints and an opt-in balancer that directs leader-dependent operations while retaining fallback behavior.Source: D. Practical Impact: Follow-on etcd… · p. 8
  • Caching and topology-aware mechanisms are complementary because they reduce requests or cross-zone traffic but do not separate reads from writes.Source: §VI · pp. 8–9

VII. CONCLUSION

The conclusion recommends routing according to the protocol work induced by each operation: direct writes to the leader, distribute reads across healthy members, and use latency to detect degradation rather than infer steady-state routing. The reported gains are strongest for degraded followers, while the steady-state measurements assume a fixed leader.

  • Latency-only routing fails to separate read-serving cost from write-forwarding cost, although it remains useful for detecting degradation.Source: §VII · p. 9
  • 64.2% lower read P99, 73.9% lower write P99, and 89.1% higher throughput are achieved under a degraded same-AZ follower.Source: §VII · p. 9
  • 29.4% lower write P50 on 3 nodes and 36.7% on 5 nodes accompany distributed linearizable reads in healthy steady-state measurements.Source: §VII · p. 9
Loading 2609.00392v1…