PaperlayerView exact revision on arXiv

Source-linked AI summary

IAPRepair: In-Network Aggregation Enhanced Proactive Repair for Erasure-Coded Storage System

Lei Liu, Yong Wang, Junqi Chen, Yangfan Liang, Qian He, Baokang Zhao

arXiv:2610.11114v1cs.DC

TL;DR

Erasure-coded repair can generate heavy cross-node traffic, while receiver capacity and heterogeneous bandwidth complicate proactive repair. IAPRepair coordinates batch construction, migration, reconstruction, and selective in-network aggregation; it reports at least 47.83% lower repair time than evaluated state-of-the-art methods.

  • Problem

    Proactive repair must coordinate migration and reconstruction despite asymmetric receiver capacity and heterogeneous node bandwidth, which can leave workloads imbalanced.

    Source: B. Problem description and Analysis · p. 5

  • Method

    IAPRepair jointly assigns reconstruction and migration tasks by load and receiver capacity, selectively aggregating reconstruction blocks that would otherwise overload receivers.

    Source: §I · pp. 1–2

  • Results

    At least 47.83% lower repair time than evaluated state-of-the-art methods is reported for IAPRepair.

    Source: §I · p. 2

  • Takeaways & Limitations

    The conclusion reports that IAPRepair significantly reduces repair time compared with existing methods in prototype testing and large-scale simulations.

    Source: §VII · p. 13

Abstract

from arXiv · show

Erasure-coded storage provides fault tolerance with substantially lower storage overhead than full replication, but repairing lost or at-risk blocks requires intensive cross-node data transfer. Existing reactive repair schemes start only after a failure, while proactive schemes can move data before failure but commonly treat migration, reconstruction, and network aggregation as loosely coupled operations. As a result, receiver?side bottlenecks, heterogeneous available bandwidth, and limited programmable-switch state continue to constrain repair paral?lelism. This paper presents IAPRepair, an in-network aggre?gation enhanced proactive repair framework for erasure-coded storage. IAPRepair jointly constructs each repair batch, assigns reconstruction providers and replacement nodes according to normalized transmission loads, schedules migration around the remaining receive capacity, and selectively enables in-network aggregation for reconstruction blocks that would otherwise over?load healthy nodes. The selective design reduces receiver-side traffic while retaining migration parallelism and respecting a configurable switch-resource budget. We implement IAPRepair with a Tofino programmable switch and 16 storage nodes, and evaluate it using both a prototype testbed and large-scale simu?lations. Across coding parameters, block sizes, node populations, and multiple STF-node scenarios, IAPRepair reduces repair time by at least 47.83% compared with the evaluated state-of-the-art methods. The results demonstrate that coordinating proactive repair decisions with in-network processing is an effective way to improve repair efficiency under bandwidth heterogeneity.

I. INTRODUCTION

IAPRepair coordinates proactive migration, reconstruction, and selective in-network aggregation to address receiver bottlenecks and heterogeneous bandwidth. It reports shorter repair times than evaluated alternatives, while its erasure-coding basis allows a block to be reconstructed from surviving blocks in its stripe.

I. INTRODUCTION

  • Existing proactive schemes combine migration and reconstruction in batches but insufficiently coordinate receiver capacity, bandwidth heterogeneity, and operation choice; INA-based repair has mainly targeted reactive repair.Applying INA to proactive repair also raises questions about which blocks use scarce switch state and how repair tasks are assigned.Source: §I · p. 1
  • Repair creates intensive cross-node traffic and concurrent tasks, while only healthy nodes can receive repaired blocks, risking receiver overload.Available bandwidth also differs across nodes, so balanced block counts may still produce imbalanced transmission times.Source: §I · p. 1
  • IAPRepair jointly assigns reconstruction providers and replacement nodes by normalized loads, then schedules migration around healthy nodes’ residual receive capacity.This couples repair decisions within each batch under heterogeneous bandwidth and receiver-capacity constraints.Source: §I · pp. 1–2
  • Selective INA targets reconstruction blocks that would otherwise bottleneck receivers, reducing receiver-side traffic while limiting switch-state use.The design retains migration parallelism and respects a configurable switch-resource budget.Source: §I · p. 2
  • At least 47.83% lower repair time than evaluated state-of-the-art methods is reported for IAPRepair.The framework was implemented with a Tofino programmable switch and 16 storage nodes, and evaluated in a prototype and simulations.Source: §I · p. 2

A. Basics of erasure coding

  • In RS(n, k), a block can be reconstructed from any other k blocks in its stripe; independent coding groups make word-level reconstruction possible.Word-level processing is the stated basis for programmable-switch repair without buffering a whole block in limited switch SRAM.Source: A. Basics of erasure coding · pp. 2–3

B. Proactive repair

Proactive repair combines migration and reconstruction before predicted failure, while in-network aggregation can reduce reconstruction traffic. IAPRepair coordinates repair assignment and selective aggregation to address receiver contention and heterogeneous bandwidth.

B. Proactive repair

  • Migration copies each block from its STF node with low transfer overhead but limited parallelism, whereas reconstruction uses healthy-node providers and supports parallel repairs at higher data-transfer cost.Coordinating both repair modes can improve repair efficiency.Source: B. Proactive repair · p. 3

C. In-network aggregation

  • In-network aggregation offloads repair-data aggregation to network devices, but programmable-switch resources are scarce and packet-arrival uncertainty can affect aggregation performance.Tofino switches support custom computation, storage, and logic during packet forwarding through P4.Source: C. In-network aggregation · p. 3

D. Related work

  • Prior proactive approaches combine migration and reconstruction but do not jointly account for receiver contention, heterogeneous available bandwidth, and limited in-network state.Programmable-network repair approaches primarily target reactive repair rather than joint scheduling of proactive migration and reconstruction.Source: D. Related work · p. 3

A. System overview

  • IAPRepair’s control plane selects reconstruction blocks, assigns migration around resulting receive load, and marks selected reconstruction traffic for aggregation by the P4 data plane.The switch forwards migration and unmarked reconstruction packets directly, while routing marked reconstruction packets through an aggregator group.Source: A. System overview · pp. 3–4, Figure 4 · p. 4

B. Problem description and Analysis

  • Only healthy nodes can receive repaired blocks, so concurrent migration and reconstruction can overload receivers and limit repair performance.A block-count-balanced schedule may still produce uneven transmission times when available node bandwidth differs.Source: B. Problem description and Analysis · p. 5
  • In-network aggregation can ease receiver demand, but batch selection also matters: uneven provider workloads can make one seven-block strategy less efficient than another.The Fig. 5 comparison limits repairs to six STF nodes in panels (a) and (b), while panel (c) repairs seven STF stripes in parallel.Source: B. Problem description and Analysis · p. 5, Figure 5 · p. 5

C. Formulation

The formulation represents migration and reconstruction assignments, node transmission loads, and selective in-network aggregation as batch decisions. Its objective is to repair more STF blocks per batch in less time while respecting provider, placement, and switch-resource constraints.

  • Reconstruction assigns healthy provider nodes and healthy replacement nodes to STF blocks, while migration assigns STF providers and healthy replacements.Each repaired block receives one designated replacement node, and blocks from the same stripe must be placed on distinct nodes.Source: C. Formulation · p. 5
  • Reconstruction requires k healthy nodes to provide data, whereas migration requires one STF node to provide the block.Source: C. Formulation · p. 5
  • A binary indicator selects whether reconstruction uses in-network aggregation, and a switch-resource constraint caps the number of aggregated blocks at θ.Source: C. Formulation · p. 6
  • The model tracks node transmission and reception volumes against available uplink and downlink bandwidth, with batch repair time determined by the node with the maximum transmission time.Without aggregation, a reconstruction destination receives k blocks; migration or aggregation-enabled reconstruction requires it to receive one block.Source: C. Formulation · p. 6
  • The objective selects migration and reconstruction sets and assignment variables to repair more STF blocks within a shorter batch time.The formulation describes this as maximizing each batch’s repair efficiency.Source: C. Formulation · p. 6

IV. DESIGN OF IAPREPAIR

IAPRepair builds proactive repair batches by selecting reconstruction tasks according to node loads, then assigning migration and replacement nodes around the remaining receive capacity. It selectively uses in-network aggregation subject to switch-resource and batch-size constraints.

IV. DESIGN OF IAPREPAIR

  • IAPRepair selects STF blocks for reconstruction, assigns their providers and replacement nodes, chooses migration blocks and destinations, then determines which reconstructions use in-network aggregation.Source: §IV · p. 6

A. Block selection for reconstruction

  • Reconstruction selection prioritizes available healthy-node uplink capacity because in-network aggregation reduces receiver traffic and makes uplink bandwidth more likely to bottleneck.Source: A. Block selection for reconstruction · p. 6
  • For each candidate reconstruction, the algorithm favors less-loaded providers using their transmission load relative to available uplink bandwidth, and selects a replacement among fault-tolerance-eligible nodes with the lowest receive load.Replacement-node loads account for the k blocks required for reconstruction, and stripe placement constraints persist across batches.Source: A. Block selection for reconstruction · p. 6
  • The number of reconstruction blocks per batch varies with transmission-load balance, while each batch is capped at 2|Hh| blocks to limit repair time and computation and cache use.The selection process stops when the maximum-to-average uplink-load ratio exceeds threshold ξ.Source: A. Block selection for reconstruction · pp. 6–7
  • In the illustrated RS(4, 2) example, the algorithm chooses the least-scored block, assigns h1 and h3 as providers, and places its reconstruction on the more idle eligible node h5.The example has one STF node and five healthy nodes, with three STF blocks already selected for reconstruction.Source: A. Block selection for reconstruction · p. 7

B. Block selection for migration

Migration selection respects the receive capacity remaining after reconstruction and favors healthy replacement nodes with the largest migration capacity. Concentrating migration blocks can reduce the number of reconstructions that need in-network aggregation.

B. Block selection for migration

  • Migration is limited by STF-node transmission capacity and by the receiving capacity left at healthy nodes after reconstruction.The section sets limits on blocks transmitted by each STF node and received by each healthy node.Source: B. Block selection for migration · p. 7
  • IAPRepair preferentially assigns migration blocks to the healthy node with the largest remaining migration receive capacity.Concentrating blocks lets bandwidth released by aggregating one reconstruction serve migration of k − 1 blocks, reducing idle bandwidth and further INA demand.Source: B. Block selection for migration · pp. 7–8
  • In the Fig. 7 example, orange placement needs INA for two reconstructions, compared with all three under green placement.The example limits each node to receiving three blocks.Source: B. Block selection for migration · pp. 7–8, Figure 7 · p. 8

3 Set nm,d

The migration-assignment procedure tracks sender and receiver limits, chooses replacement nodes with the greatest remaining receive capacity, and admits only blocks that fit both limits.

3 Set nm,d

  • The procedure initializes migration transmission and reception counts to zero, then computes their per-node limits before assignment.The limits are derived using Equations 19 and 20.Source: §3 · p. 8
  • For each STF block, the procedure selects a fault-tolerant healthy replacement node with the largest remaining migration receive capacity.The selected block is recorded for that replacement node.Source: §3 · p. 8
  • Replacement-node groups are traversed in descending order by group size, and a migration is admitted only if its sender and receiver remain below their limits.Admitted blocks are added to the migration set and removed from the STF-block set.Source: §3 · p. 8

C. In-network aggregation decision

IAPRepair selectively assigns in-network aggregation to reconstruction blocks to address receiver-capacity demand while respecting programmable-switch resource constraints.

C. In-network aggregation decision

  • A node’s INA-assigned reconstruction count must meet its calculated threshold to eliminate the repair-performance limitation imposed by receiving capacity.This threshold is derived from the receive-capacity constraint.Source: C. In-network aggregation decision · p. 8
  • The INA assignment may be infeasible under switch resource constraints, so IAPRepair seeks to minimize the maximum difference between node demand and assigned INA blocks.The selection is constrained by programmable-switch internal resources.Source: C. In-network aggregation decision · p. 8
  • The procedure computes each healthy node’s INA demand, then repeatedly selects a block for the node with the largest unmet demand until requirements are met or no more blocks can use INA.A block is selected from those assigned to that node for replacement.Source: C. In-network aggregation decision · p. 8

D. Algorithm Analysis

The analysis gives complexity bounds for reconstruction-block selection and describes the traversal and per-selection cost of migration assignment.

D. Algorithm Analysis

  • Algorithm 1 selects at most 2|Hh| reconstruction blocks, traverses S at most 2|Hh| times, and computes each block’s score in O(|Hh|).Its stated time complexity is O(2|Hh|^2|S|).Source: D. Algorithm Analysis · p. 8
  • Algorithm 2 traverses S twice; its first traversal selects a replacement node for each STF block at O(|Hh|) per selection.The first traversal therefore makes |S| replacement-node selections.Source: D. Algorithm Analysis · p. 8

1 Set nina

Algorithm 3 selects a reconstruction block assigned to a chosen healthy replacement node and adds it to the INA set. Its cost scales with the INA block limit and the number of healthy nodes; IAPRepair also extends to LRCs.

1 Set nina

  • A selected STF block assigned to the chosen healthy node is added to S_ina as its replacement.For LRCs, IAPRepair prioritizes migrating global parity blocks because reconstructing a local block requires substantially less data than reconstructing a global parity block.Source: §2 · p. 9
  • Algorithm 3 runs in O(θ|Hh|), and the sequential algorithms have overall complexity approximately O(2|Hh|^2|S| + θ|Hh|).Algorithm 2 takes O(|S|(|Hh|+1)); Algorithm 3 adds at most θ block selections.Source: §2 · p. 9

V. IMPLEMENTATION

The prototype implements IAPRepair’s repair algorithms in Python and its programmable-switch processing in P4. The switch forwards ordinary packets and aggregates only marked reconstruction traffic.

V. IMPLEMENTATION

  • The prototype uses C++, Python, and P4, with Linux sockets for repair traffic and Jerasure for decoding coefficients and Galois-field operations.It adopts Paint’s ACP protocol, which extends UDP with retransmission and switch-resource-management fields.Source: §V · p. 9
  • The programmable switch parses packet types, sends marked reconstruction packets to aggregators, and forwards other packets directly.Migration data flows from STF nodes to healthy nodes; healthy nodes also send reconstruction data and receive both kinds of repair data.Source: §V · p. 9, Figure 8 · p. 9
  • IAPRepair assigns each STF block reconstructed through INA a group of aggregators, supporting continuous sender transmission and buffering packet-arrival differences.A group of Na aggregators lets a sender transmit at least Na packets consecutively without awaiting receiver acknowledgments.Source: §V · p. 9
  • Each aggregator XORs incoming packet data into an intermediate result and forwards it when the received-packet count reaches the configured flow count.The aggregator then resets the intermediate result.Source: §V · pp. 9–10
  • Packet identifiers select the aggregator group and instance, while sequence offsets handle out-of-order data and version checks detect mismatched aggregation.Detected transmission errors require retransmission.Source: §V · p. 10

VI. EVALUATION

The evaluation combines a 16-storage-node Tofino prototype with simulations and compares IAPRepair with FastPR, SelectiveEC, and Migration. Across the tested repair scenarios, IAPRepair consistently reduces repair time, including under varying coding parameters, node counts, and STF-node counts.

A. Experimental settings

  • The prototype uses a Tofino switch and 16 storage nodes, and repair time includes data transfer plus algorithm runtime.Unless otherwise specified, tests average 10 runs using RS(9, 6), 8 MB blocks, ξ=0.9, 512 stripes, and one STF node; node bandwidths are configured from heterogeneous workload measurements.Source: §VI · p. 10, A. Experimental settings · p. 10

B. Repair time

  • 47.83% to 68.04% shorter repair time than SelectiveEC, 53.79% to 65.49% than FastPR, and 57.76% to 75.94% than Migration across four RS coding parameters.Fig. 9 presents repair time for RS(6, 4), RS(9, 6), RS(12, 8), and RS(14, 10).Source: Figure 9 · p. 10, B. Repair time · p. 10
  • As k increases, reconstruction needs more data; IAPRepair avoids SelectiveEC’s degradation near RS(14, 10) by reducing transmission volume through INA.Across 2 MB to 32 MB blocks, all methods take longer as blocks grow, while IAPRepair remains fastest, with average repair time 55.31%, 61.97%, and 67.87% shorter than SelectiveEC, FastPR, and Migration, respectively.Source: B. Repair time · p. 10
  • As storage-node count grows, IAPRepair repair time decreases smoothly and is at least 40.97%, 47.82%, and 60.48% shorter than SelectiveEC, FastPR, and Migration, respectively.More nodes enable concurrent reconstruction; Migration does not improve because of the STF node’s uplink limit, while FastPR’s curve is step-like.Source: B. Repair time · pp. 10–11
  • With one, two, and three STF nodes, IAPRepair reduces repair time by 53.84%, 49.09%, and 36.79%, respectively, versus the baseline methods.Its repair time rises with STF-node count but remains lower than the compared methods.Source: B. Repair time · p. 11, Figure 12 · p. 11
  • Repair time is shortest when ξ is 0.9 to 0.95, while larger ξ increases batch time and can make measured network conditions obsolete.The experiments set ξ to 0.9.Source: B. Repair time · p. 11, Figure 13 · p. 11, Figure 14 · p. 12
  • IAPRepair completes repair even if an STF node fails during the process, after which it can only reconstruct; failure timing has greater impact at smaller k.For RS(6, 4), RS(9, 6), RS(12, 8), and RS(14, 10), failure at 10 seconds reduces repair time by 22.56%, 22.31%, 21.97%, and 20.53%, respectively, compared with failure at 1 second.Source: B. Repair time · p. 12, Figure 15 · p. 12
  • On LRC(6, 4, 2), LRC(8, 6, 2), and LRC(10, 8, 2), IAPRepair reduces repair time by 29.79%–69.65% versus baseline methods.The evaluation uses these LRC configurations to test generality across erasure codes.Source: B. Repair time · p. 12, Figure 16 · p. 12

C. Repair Overhead

The overhead evaluation reports switch-resource use and algorithm runtime, while the conclusion attributes repair-time improvements to STF-stripe partitioning and INA. The switch’s SRAM use varies with the number of INA-reconstructed blocks.

C. Repair Overhead

  • Across coding parameters, the maximum INA-reconstructed STF-block count is 21, 29, 27, and 30 for RS(6, 4), RS(9, 6), RS(12, 8), and RS(14, 10), respectively.With 32 aggregator groups of 64 aggregators, SRAM is distributed over 11 stages, each below 15.00% utilization; TCAM overhead is negligible.Source: C. Repair Overhead · p. 12, Table II · p. 13, Figure 18 · p. 13
  • IAPRepair runtime is at least 17.71% lower than SelectiveEC and FastPR across node counts, and at least 18.18% lower across STF-block counts.The evaluation varies storage nodes from 20 to 100 and STF blocks from 200 to 1000; the corresponding tables report these runtime comparisons.Source: C. Repair Overhead · pp. 12–13, Table III · p. 13, Table IV · p. 13
  • IAPRepair’s computational overhead is less than 1 second and can be hidden within data transfer for the previous subset in pipelined repair.Production nodes may contain more than 1.5 million blocks, which can be partitioned and repaired sequentially.Source: C. Repair Overhead · p. 13

VII. CONCLUSION

  • The conclusion attributes improved proactive-repair efficiency to rational STF-stripe partitioning and INA for reconstructing STF blocks.The paper reports support from prototype testing and large-scale simulations.Source: §VII · p. 13
Loading 2610.11114v1…