PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive Switching Between Leader-Based and Leaderless BFT Protocols

Sudip Bhujel, Yue Li, Ning Zhang, Y. Thomas Hou, Wenjing Lou, Yang Xiao

arXiv:2609.31388v1cs.DC

TL;DR

Static BFT protocols face changing network conditions, while switching across protocol families must preserve the committed log and keep handoff costs low. BFTIDE embeds transition coordination in ongoing BFT operation and uses a policy to propose protocols; experiments report lower latency under adverse conditions than static protocols and BFTBrain. Its switching guarantee still depends on incumbent progress.

  • Problem

    Static protocol configurations are suboptimal as network conditions change, and switching protocols must preserve a common committed log while keeping transition overhead low.

    Source: §1 · pp. 1–2

  • Method

    BFTIDE embeds switch coordination in incumbent BFT operation and uses an offline-trained reinforcement-learning policy to propose protocols from system metrics.

    Source: §1 · p. 2, §8 · p. 12

  • Results

    BFTIDE reports lower latency than static BFT protocols and BFTBrain under varied network delays, with 3.9 s median latency versus 6.3 s for HotStuff and 26.4 s for BFTBrain at 120 KB/s.

    Source: §1 · p. 2, §8 · p. 12

  • Takeaways & Limitations

    BFTIDE demonstrates protocol adaptation across leader-based partially synchronous and leaderless asynchronous protocols while BFT-SMR operation continues.

    Source: §1 · p. 2, §8 · p. 12

    BFTIDE cannot guarantee escape from an incumbent that has stopped committing because switching requires incumbent progress through the boundary.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Byzantine fault-tolerant (BFT) protocols are known for providing operational consistency and resilience in distributed systems. However, evolving network conditions, often driven by the network's inherent dynamism or adversarial influence, make it suboptimal to rely on a static protocol at all times. Existing BFT protocol adaptation solutions switch only among leader-based protocols and coordinate each switch through a separate consensus round, leaving them ineffective at handling severe asynchrony or situations in which an adaptive adversary targets the network's leader. We propose BFTide, a protocol adaptation architecture that enables a BFT system to intelligently and swiftly switch to a suitable protocol as network conditions shift. BFTide integrates a novel protocol switching layer that embeds protocol transition logic into the ongoing BFT operation, enabling safe and low-overhead transitions between partially synchronous leader-based protocols and asynchronous leaderless protocols. It further incorporates an offline-trained reinforcement learning policy that allows nodes to propose protocols at runtime based on observed system metrics. Experimental results show that BFTide reduces transaction latency under adverse network conditions compared with static BFT protocols and the state-of-the-art BFT protocol adaptation scheme BFTBrain (NSDI'25), while maintaining comparable throughput. The switching layer adds a modest 10-21% overhead to median latency when idle and requires no separate consensus round per switch.

1 Introduction

BFTIDE adapts between partially synchronous leader-based and asynchronous leaderless protocols while ongoing operation continues, separating protocol proposals from transition coordination. Its experiments report lower latency than static protocols and BFTBrain under adverse conditions, with comparable HotStuff performance under benign conditions.

Adapting to Evolving Network Conditions

  • Leader-based protocols can suffer leader bottlenecks under targeted delay, while asynchronous protocols incur substantial computation and communication overhead.Prior adaptation schemes primarily target synchronous or partially synchronous protocols and lack a clear strategy for asynchronous conditions.Source: §1 · pp. 1–2

Contributions

  • BFTIDE switches between leader-based partially synchronous and leaderless asynchronous protocols without interrupting continuous operation.The architecture separates transition coordination from the policy that proposes when and where to switch.Source: §1 · p. 2
  • The switch layer embeds agreement in incumbent messaging and consensus, using signed votes and a certificate to coordinate the target and transition boundary.Replicas verify the evidence and reach the agreed boundary before activating the target; switching progress remains subject to the incumbent’s liveness conditions, and safety requires an additional certificate-delivery condition.Source: §1 · p. 2
  • An offline-trained reinforcement-learning policy uses system measurements to propose protocols, while the switch layer governs activation.The policy is formulated as an MDP and instantiated with a DQN; its reward accounts for performance and switching cost.Source: §1 · p. 2

Implementation and Validation

  • At 120 KB/s, BFTIDE’s median latency is 3.9 s versus 6.3 s for HotStuff and 26.4 s for BFTBrain.With 250 ms injected leader delay in a 31-node system, BFTIDE tracks the better-performing protocol across conditions; averaged across four conditions at 120 KB/s, its latency is about 70% lower than BFTBrain’s.Source: §1 · p. 2

2 Background and Problem Description

BFT protocols differ in their synchrony assumptions and progress mechanisms, creating trade-offs between leader-based and asynchronous designs. Switching among them must coordinate the target and transition point while preserving BFT-SMR safety and liveness.

2.1 BFT Protocols

  • BFT-SMR requires non-faulty nodes to agree on transactions and execute them in the same order, while eventually committing client transactions.Agreement and total order together are safety.Source: §2.1 · pp. 2–3
  • Partially synchronous leader-based protocols use leaders and view changes, while asynchronous leaderless protocols tolerate arbitrary message delays but rely on costly mechanisms for liveness.HotStuff’s progress depends on a responsive leader; ACS protocols use reliable broadcast and agreement to decide a common subset.Source: §2.1 · p. 3

2.2 Challenges for BFT Protocol Switching

  • Protocol switching must secure swift agreement on the change, reconcile different execution structures, and let nodes propose choices suited to observed network conditions.Transitions must preserve safety and liveness despite differences such as ACS’s canonical ordering of accepted requests and PBFT variants’ leader-assigned ordering.Source: §2.2 · p. 3

3 The BFTIDE Architecture

BFTIDE separates protocol selection from enforcement and coordinates transitions through the incumbent protocol’s operation. Its switch certificates identify a target and boundary, and handoff rules preserve the committed state and pending requests when moving between HotStuff and FIN.

3 The BFTIDE Architecture

  • The Protocol Switch Layer finalizes the selected protocol independently of incumbent consensus and policy proposal generation.BFTIDE assumes fixed membership, authenticated channels, at most f Byzantine replicas among n = 3f + 1, and eventual message delivery for retransmitted messages; liveness remains subject to the active protocol’s assumptions.Source: §3 · p. 4, §3.1 · p. 4

3.1 Network Model

  • Switching requires incumbent progress through the certificate’s boundary, so BFTIDE cannot guarantee escape from an incumbent that has stopped committing.A timeout or policy proposal alone does not authorize activation; persistent loss of quorum communication is outside the liveness conditions.Source: §3.1 · p. 4

3.2 Protocol Switch Layer Functions

  • Replicas embed signed switch votes in incumbent messages, and the certificate fixes a target and log boundary without a separate consensus round.The target continues from the incumbent’s application state at that boundary.Source: §3.2 · p. 4

3.2.1 Windows, Metrics, and Switch Votes

  • Nodes aggregate at least 2f+1 reports and vote only after the policy proposes the same target in consecutive windows; a dwell period helps prevent thrashing.The policy uses the previous window’s aggregate, and the dwell period doubles after a rapid re-switch.Source: §3.2.1 · p. 5

3.2.2 Switch Certificate

  • A certificate forms from 2f+1 distinct signed votes, and quorum intersection ensures at most one target can be certified for a window.Votes with different metric digests do not combine; without a valid certificate, honest replicas retain the incumbent.Source: §3.2.2 · p. 5, §3.2.1 · p. 5

3.2.3 Handoff

  • Handoff activation requires a verified certificate and commitment through its exact boundary; the transition retains application state and re-queues uncommitted requests.Stopping the incumbent safely also requires timely delivery of the certificate, an additional premise not guaranteed by partial synchrony or FIN’s eventual delivery for fixed k.Source: §3.2.3 · p. 5

3.2.4 Instantiation for HotStuff and FIN

  • HotStuff-to-FIN switching uses a block-height boundary, while FIN-to-HotStuff switching uses an epoch boundary and requires deterministic execution of the terminal subset.The HotStuff leader cannot withhold the switch decision; requests omitted from FIN’s selected subset remain eligible for re-proposal under HotStuff.Source: §3.2.4 · p. 6

4 Learning to Select Protocols

BFTIDE uses an offline-trained policy to propose protocols from system metrics, while the incumbent consensus layer certifies a shared switch boundary without a separate consensus round.

4 Learning to Select Protocols

  • BFTIDE trains its DQN offline in a simulator calibrated with testbed latency and throughput, then uses runtime metrics to propose protocols.The calibration covers each candidate protocol under each network condition.Source: §4 · p. 6, §4.2 · p. 7

4.1 MDP Formulation for Protocol Selection

  • The MDP state combines six performance, network, and incumbent-protocol dimensions, while its reward balances throughput, latency, HotStuff delay penalties, and switching cost.The delay penalty applies to HotStuff, and the switch indicator discourages excessive switching.Source: §4.1 · pp. 6–7

4.2 Instantiation with DQN

  • DQN maps continuous performance inputs to discrete HotStuff or FIN proposals and minimizes Huber temporal-difference loss, allowing expected future gains to outweigh transient switch latency.The target value combines observed reward with the discounted highest next-state action value.Source: §4.2 · p. 7

4.3 Metrics Synchronization

  • Replicas exchange authenticated window reports through ordinary protocol messages; 2f+1 reports support agreed aggregates and switch votes bound to their digest.The digest excludes peer delays and delayed-peer fraction, while offered load remains replica-specific.Source: §4.3 · pp. 7–8

5 Analysis

The analysis gives conditional safety arguments for agreement, total order, and at-most-once execution across handoffs, alongside liveness conditions and limits on what the handoff can recover.

5 Analysis

  • Under n = 3f+1 and the stated timing premise, 2f+1-signature certificates agree on one target and boundary through an honest signer in every certificate intersection.The timing premise requires at least f+1 honest replicas to receive the certificate before supporting incumbent decisions above the boundary.Source: §5 · p. 8
  • The target extends the committed prefix in a deterministic order, and induction over handoffs preserves global order; matching local round numbers alone is insufficient.FIN orders agreed payloads deterministically, while HotStuff supplies ordered blocks.Source: §5 · p. 8
  • At-most-once execution holds across a handoff when the target inherits deduplication state and re-proposed requests retain their client identity and sequence number.Eventual inclusion of uncommitted requests additionally requires continued dissemination and the target's inclusion guarantees.Source: §5 · p. 8
  • Replicas can complete a handoff if the incumbent reaches the boundary, honest replicas receive the certificate, and the prefix remains available for verified catch-up.Progress then depends on the target's liveness conditions with n−f honest replicas in the same target instance; activation need not be simultaneous.Source: §5 · p. 8
  • Poor policy choices or repeated switching can impair performance without bypassing safety checks, and BFTIDE does not claim to escape an incumbent stalled before the boundary.Dissemination or catch-up may also delay activation.Source: §5 · p. 8
  • With at least 2f+1 reports, coordinate-wise medians for common report coordinates lie within the honest range, but global accuracy is not guaranteed and Byzantine bias may persist across windows.The f+1 retention threshold alone does not provide the stated bound for per-peer delay coordinates.Source: §5 · p. 8

6 Implementation and Evaluation

BFTIDE was implemented and evaluated as a runtime switch between HotStuff and FIN under changing network conditions. Across offered loads and network scenarios, it reduces latency versus BFTBrain on average, with measurable idle overhead and sensitivity to switching parameters.

6 Implementation and Evaluation

  • The prototype implements PBFT, HotStuff, Dumbo, and FIN, but the adaptation experiments restrict switching to HotStuff and FIN.Its three-layer DQN has two 64-unit hidden layers and is trained offline in a simulator calibrated to testbed latency and throughput.Source: §6 · pp. 8–9

6.1 Evaluation Setting

  • The evaluation uses independently generated open-loop Poisson requests and varies offered load and emulated network delays; latency includes queuing and consensus.The primary configuration uses 31 replicas, 250-byte transactions, and a five-round window; n=4 and n=10 configurations support benchmarks and sensitivity tests.Source: §6.1 · p. 9

6.2 Benchmarking Candidate Protocols

  • At n=10, HotStuff has the lowest latency and exceeds 145 KB/s throughput; at n=4, PBFT has the lowest latency.The benchmarks compare HotStuff, FIN, PBFT, and Dumbo across increasing batch rates in the safe network condition.Source: Figure 3 · p. 9, §6.2 · p. 9

6.4 Evolving Network Conditions

  • Across evolving delay phases, BFTIDE switches between protocols, while the unresponsive-replica run shows no clear latency benefit.In the LAN run, it returns to HotStuff after the leader attack ends. Under emulated geography, regional delay roughly triples HotStuff’s median latency but barely affects FIN; the table reports median latency with and without a 250 ms leader delay.Source: Figure 5 · p. 10, §6.4 · p. 10, Table 1 · p. 10

6.5 Performance Under Varying Offered Load

  • At 120 KB/s, BFTIDE averages 2.51 s latency versus BFTBrain’s 8.51 s, a reduction of about 70%, while average throughput reaches 110.9 KB/s.Across the four scenario-level averages, BFTIDE has the lowest average latency from 40 to 120 KB/s; HotStuff is slightly lower at 20 KB/s. At 120 KB/s, HotStuff averages 115.6 KB/s throughput.Source: §6.5 · p. 11, Figure 7 · p. 11

6.6 Protocol Switching Overhead

  • Without switching, the layer adds 21% to HotStuff’s median latency and 10% to FIN’s, with throughput unchanged for HotStuff and within 5% for FIN.Reports piggyback on incumbent messages, while separate probes and acknowledgements remain; inference takes 0.18 ± 0.02 ms.Source: Table 2 · p. 11, §6.6 · p. 11

6.7 Window Size and Threshold Sensitivity

  • Window size trades reaction time against switching frequency, while a threshold above the detectable round-trip excess can cause intermittent detection and policy oscillation.With a 100 ms leader delay, the first switch occurs at round 30 for w=5 and round 60 for w=10; the paper fixes w=5 and T=250 ms for n=31 except in specified experiments.Source: Figure 8 · p. 12, §6.7 · p. 12

7 Related Work

Prior BFT adaptation work spans dual-path designs and protocol switching, but BFTIDE is positioned against approaches with tuned triggers or narrower protocol portfolios. The closest comparison is BFTBrain, while configuration adaptation is treated as complementary work.

Dual-path BFT

  • Dual-path schemes pair optimistic and pessimistic protocols, but require tuned transition triggers unless clients choose finality; BFTIDE instead uses learning-based decisions.Parallel variants run both paths simultaneously, which is inefficient when pessimistic conditions are rare.Source: §7 · p. 12

Multi-protocol Switching

  • Abstract and ADAPT support protocol switching through a unified composition framework or a quality-control module observing performance under changing network conditions.Abstract introduces AZyzzyva and Aliph as candidate protocols.Source: §7 · p. 12
  • BFTBrain targets synchronous and partially synchronous conditions with an RL-selected leader-based portfolio, while BFTIDE adds leaderless protocols and in-band switching.BFTBrain’s throughput-prioritized reward and separate consensus round for switching also distinguish it from BFTIDE.Source: §7 · p. 12

Complementary BFT Adaptation

  • Configuration-adaptation systems adjust voting weights, leader placement, quorum sizes, or replica roles, whereas BFTIDE switches protocols and leaves inner-protocol tuning for future work.These complementary approaches also rely on consensus-coordinated adaptation.Source: §7 · p. 12

8 Conclusion

BFTIDE enables BFT-SMR systems to switch protocols as network conditions evolve, including under adversarial delay attacks. Its switching mechanism and offline-trained policy are reported to improve long-term performance under varied network delays.

Conclusion

  • BFTIDE switches between leader-based partially synchronous and leaderless asynchronous protocols without interrupting SMR operation.Switching remains subject to incumbent progress; an offline-trained reinforcement-learning policy processes performance metrics to propose protocols.Source: §8 · p. 12

A Further Discussion and Limitations

The discussion identifies boundaries in metric robustness, protocol coverage, scale, and policy-training transfer. It also points to a representative-protocol schematic and describes the scope of the DQN architecture and training section.

A Further Discussion and Limitations

  • Repeated bias within the honest-report range may still affect protocol-proposal quality despite the median’s bound on Byzantine influence.The aggregation keeps each metric coordinate with at least 2f+1 observations within the range of included honest reports under the stated assumptions.Source: A Further Discussion and Limitations · p. 15
  • The evaluation restricts the protocol pool to HotStuff and FIN; adding protocols, including DAG designs, requires the switch-certificate interface and validation against each commit rule.Source: A Further Discussion and Limitations · p. 15
  • The evaluation provides a limited scale comparison: its primary setting is n=31, supplemented by n=4 and n=10 experiments, with timeout settings fixed per experiment.The experiments exercise switching, metric synchronization, and delay-regime adaptation under controlled adverse conditions.Source: A Further Discussion and Limitations · p. 15
  • Changes in replica count or geographic placement can mismatch policy-training conditions; representative offline retraining may mitigate this, but cross-deployment effectiveness remains unevaluated.Source: A Further Discussion and Limitations · p. 15

B Schematic of Representative BFT Protocols

C DQN Architecture and Training

C.1 Network Architecture

The DQN takes six system metrics and protocol-state variables as input and estimates a Q-value for each candidate protocol. Its feedforward network uses two 64-unit hidden layers with ReLU activations.

Network Architecture

  • Six inputs—latency, throughput, delay indicator, offered load, incumbent protocol, and delayed-peer fraction—feed the network, whose outputs estimate Q-values for HotStuff and FIN.Source: §C.1 · p. 15
  • Figure 9 compares representative BFT protocols in one consensus round, excluding client request handling.Source: Figure 9 · p. 16
  • The feedforward network has two hidden layers of 64 units with ReLU activations and 4,738 parameters.Source: §C.1 · p. 16

C.2 Offline Training Environment

The offline training environment uses a simulator calibrated from the 31-replica testbed across specified network conditions. It normalizes its inputs and models changing conditions, load, and reward across training episodes.

C.2 Offline Training Environment

  • The simulator uses measured baseline latency and achieved-throughput fractions across undelayed, leader-delay, global-delay, and partition conditions; the 200 ms leader-delay value is interpolated.All delay values are one-way; the partition case is modelled.Source: §C.2 · p. 16
  • Latency, throughput, and offered load are min-max normalized and clipped to [0,1]; network indicators already lie in that range, and the incumbent is encoded by protocol index.Training and inference share the normalization routine.Source: §C.2 · p. 16
  • The simulator models delayed-peer fractions by condition for T=250 ms, with a 0.05 probability of marking one peer delayed when none are nominally delayed.The indicator p_t is set when at least f+1 peers are delayed.Source: §C.2 · p. 16
  • Each 64-window episode samples initial protocol, offered load, and network conditions, then resamples conditions and perturbs latency and throughput with Gaussian noise.The reward uses α_tp=1, α_l=1, α_ld=0.5, α_s=1, and α_sw=0.5, and is clipped to [−2,2] during training.Source: §C.2 · p. 16

C.3 Offline Training Configurations and Combined Workflow with Inference

The training configuration specifies how the DQN is optimized and how replicas use the trained checkpoint at runtime. The section also identifies an earlier checkpoint used for one varying-load comparison.

C.3 Offline Training Configurations and Combined Workflow with Inference

  • Training minimizes Huber loss with Adam, uniformly sampled experience replay, a periodically copied target network, and exponentially decaying ε-greedy exploration.Training takes 118 s for 40,000 environment steps with seed 42; inference takes 0.18 ms.Source: §C.3 · p. 16
  • At inference, replicas load the trained checkpoint, keep its parameters fixed, and propose the protocol with the highest Q-value; Figure 10 depicts the combined training and inference workflow.The supplied Figure 11 caption identifies latency-versus-replica-count plots under leader delay for HotStuff and FIN.Source: §C.3 · p. 16, Figure 10 · p. 17, Figure 11 · p. 17
  • The varying-load comparison uses an earlier checkpoint calibrated only with the 10 ms global-delay case, before later calibration additions.The later calibration added 25–100 ms global-delay cases and an interpolated 200 ms leader-delay case.Source: §C.3 · p. 16

D Additional Evaluation

The additional evaluation examines cluster scaling, behavior with unresponsive replicas, replica activation timing, and how network measurements shape policy proposals.

D.1 Scaling the Cluster

  • At 31 replicas, HotStuff remains faster than FIN at 100 ms leader delay (1.99 s against 2.99 s), while FIN is faster at 250 ms (3.21 s against 6.30 s).At 4 and 10 replicas, FIN is already faster at 100 ms; the crossover shifts with cluster size, while FIN latency is flat across leader-delay values.Source: §D.1 · p. 17

D.3 Protocol Activation across Replicas

  • For the switch to FIN, all 31 replicas record entry within approximately 0.81 s (median lag 243 ms); switching back, 30 enter within 0.3 s (median 29 ms), while one enters after 1.20 s.Figure 12 measures switch-event lag relative to the first replica; timestamps mark entry into the switch routine, before target initialization.Source: Figure 12 · p. 17, §D.3 · p. 17

D.2 Unresponsive Replica Measurements

  • BFTIDE switches to FIN at round 135 after treating unresponsive replicas as delayed; all 21 responsive replicas reach round 200, while ten stop at round 100.HotStuff records 1,158 view timeouts; its plotted window-p50 mean rises from 1.7 s before faults to 4.8 s during faults, while FIN and BFTIDE average 5.7 s during the fault period. The traces use different aggregation and characterize switching behavior rather than directly comparable cluster latencies.Source: §D.2 · p. 17

D.4 Network Measurements and Policy Decisions

  • Individual-peer delay counts exposed leader delay even while median probe RTT stayed below threshold; median aggregation remains within honest reports under the stated fault assumptions.Bias within that honest-report range can still affect proposals.Source: Figure 13 · p. 18, §D.4 · p. 18
  • With HotStuff active, the policy retains it over a wider delayed-peer range at low latency, but selects FIN above an approximately 0.2 delayed-peer fraction; with FIN active, HotStuff proposals occupy a narrow region.The FIN-to-HotStuff proposal region is near zero delayed peers and intermediate normalized latencies, showing that proposals depend on the incumbent as well as measured conditions.Source: Figure 14 · p. 18, §D.4 · p. 18

D.3 Protocol Activation across Replicas

  • Different activation times preserve a common committed history when lagging replicas verify the switch certificate and obtain and execute missing committed data before activating the target.The incumbent cannot extend the agreed prefix beyond its terminal height, and the target must extend that same prefix; agreement on terminal height alone is insufficient.Source: §D.3 · pp. 17–18
Loading 2609.31388v1…