PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Conventional Federated Learning via High-Order Regularization

Alireza Kabgani, Masoud Ahookhosh

arXiv:2609.09904v1cs.LGstat.ML

TL;DR

HiFedProx addresses unequal federated client displacements by replacing FedProx’s quadratic penalty with a scale-matched power regularizer and a finite-budget procedure with Armijo backtracking. In paired FEMNIST experiments, intermediate exponents improved stressed performance, while larger exponents reduced displacement tails at higher local cost, making p=5–7 the most useful range.

  • Problem

    Multiple local updates can produce highly unequal client displacements, motivating regularization that remains mild for typical movements while responding more strongly to unusually large ones.

  • Method

    HiFedProx uses scale-matched power-type client regularizers whose exponent controls response shape while preserving the regularization-gradient magnitude at reference displacement R, combined with finite-budget optimization and same-minibatch Armijo backtracking.

  • Results

    p=7 gives the lowest moderate-stress loss and p=6 the lowest severe-stress loss, while p=8 further reduces displacement tails without improving predictive performance and increases Armijo trial cost.

  • Takeaways & Limitations

    The exponent should be calibrated rather than maximized; in the present experiment, p=5–7 provides the most useful range.

  • Takeaways & Limitations

    The evaluation is limited to one 60-writer subset, one architecture, five seeds, and one composite stress intervention.

Abstract

from arXiv · show

Federated clients that perform several local optimization steps can return parameter displacements with widely different magnitudes. The quadratic regularization of FedProx grows linearly with displacement and therefore offers limited control over the contrast between ordinary and unusually large client movements. We here introduce HiFedProx, which replaces the quadratic penalty with a scale-matched power-type regularizer indexed by $p\geq2$. All powers have the same regularization-gradient magnitude at a reference displacement $R$, while every $p>2$ gives a weaker response below $R$ and a stronger response above it. An exact affine reference calculation shows that increasing $p$ compresses relative displacement disparities, although very large powers approach fixed-radius behavior and increase local curvature. HiFedProx combines this geometry with finite-budget stochastic client optimization and same-minibatch Armijo backtracking. In paired five-seed experiments on a frozen 60-writer FEMNIST subset, a common-parameter study over $p\in\{2,3,4,5,6,7,8\}$ shows similar clean-training performance but substantial gains under composite stress. The lowest moderate- and severe-stress losses occur at $p=7$ and $p=6$, improving over $p=2$ by $11.44\%$ and $23.16\%$, respectively. Although displacement-tail ratios continue to decrease through $p=8$, predictive performance peaks in an intermediate range and Armijo trial cost increases with $p$. These results indicate that the exponent should be calibrated rather than maximized. In our experiments, $p=5$--$7$ provides the most useful range.

I. INTRODUCTION

The paper targets unequal client displacements in federated learning by replacing FedProx’s fixed quadratic response with scale-matched high-order regularization. HiFedProx preserves a first-order finite-budget procedure while using the exponent p to control how strongly unusually large movements are penalized.

  • Motivation: Unequal client displacements arise from heterogeneous data, aggressive local optimization, and different local workloads, motivating explicit control of update magnitudes.
  • Motivation: High-order regularization makes the response milder below R and stronger above R than the scale-matched quadratic penalty.The normalization gives every p the same regularization-gradient magnitude at displacement R, while p controls the response shape.
  • Method: HiFedProx combines the power-type regularizer with finite-budget stochastic client optimization and same-minibatch Armijo backtracking.All exponents use the same initialization, sampling, local optimizer, aggregation, and gradient budget; only the radial regularization term changes.
  • Study design: The study evaluates p ∈ {2, 3, 4, 5, 6, 7, 8} under matched gradient and communication budgets to identify a useful exponent range.

II. RELATED WORK

The paper extends proximal federated optimization beyond methods that only vary a quadratic coefficient. It remains distinct from complementary drift, variance, sparse-regularization, and second-order approaches by changing the regularizer’s power while retaining a first-order procedure.

  • Proximal methods: HiFedProx differs from FedProx and adaptive proximal methods by adding exponent p as a design parameter instead of only adjusting a quadratic coefficient.Its regularizers are calibrated to share the same gradient magnitude at reference displacement R.
  • Complementary methods: SCAFFOLD, FedNova, and FedVARP address client drift, heterogeneous local computation, and partial-participation variance, making them complementary to high-order proximal regularization.
  • Second-order and clipping methods: Unlike federated second-order methods using Hessian information, HiFedProx modifies the client objective while remaining entirely first-order.Server-side clipping acts after local training, whereas high-order regularization influences the optimization trajectory during local training.

III. SCALE-MATCHED REGULARIZATION AND HIFEDPROX

The method replaces the quadratic proximal term with a scale-matched power regularizer whose reference gradient strength is fixed while p controls its radial shape. The crossover R is not a clipping radius or hard constraint.

  • A. Federated objective and scale matching: All powers have regularization-gradient magnitude µR at ∥d∥ = R, making R the crossover where their normalized responses agree.The figure depicts normalized quadratic and pth-order gradient magnitudes, both equal to one at ∥d∥/R = 1.
  • A. Federated objective and scale matching: For p > 2, the regularizer responds more weakly below R and more strongly above R than the scale-matched quadratic model.
  • A. Federated objective and scale matching: The exponent p controls response shape, µ rescales it, and R sets the crossover scale rather than imposing clipping or a hard constraint.

B. Finite-budget client update

Each selected client starts from the global model, performs finite-budget stochastic updates on regularized minibatch objectives, and uses same-minibatch Armijo backtracking before returning its displacement. The procedure adds forward evaluations but no extra backward passes or communication.

  • B. Finite-budget client update: Each selected client initializes at w_t, samples minibatches, performs K_t local iterations, and returns u_t − w_t after finite-budget optimization.The algorithm applies regularized stochastic directions and returns the final displacement from the current global model.
  • B. Finite-budget client update: Armijo backtracking evaluates the current and trial objective values on the same minibatch and retains the current iterate if no trial passes.
  • B. Finite-budget client update: Each local iteration uses one backward pass, while every Armijo trial adds one forward evaluation without adding communication.

IV. GEOMETRY AND LOCAL SAFEGUARD

The affine reference calculation shows that higher powers compress relative client-displacement disparities while preserving ordering, but very large powers approach fixed-radius behavior. Curvature grows outside the crossover, motivating Armijo backtracking as a finite-budget safeguard.

  • Affine client-model response: Higher powers preserve displacement ordering while compressing relative disparities, with stronger compression as p increases.The transformation is selective rather than uniform: smaller-than-reference displacements increase, while larger ones decrease relative to the quadratic response.
  • Affine client-model response: At θ = 1, every power produces the common displacement R, establishing the crossover where high-order and quadratic responses agree.
  • Large-power limit: Very large powers approach a fixed-radius step, suppressing extreme movements but potentially erasing useful magnitude information.This limiting behavior cautions against treating larger exponents as universally better.
  • Local safeguard: For D > R, regularizer curvature grows rapidly with p, so larger powers can require smaller accepted steps and more backtracking.For fixed D < R, the Lipschitz bound tends to zero as p increases, although moderate-power curvature need not change monotonically.
  • Local safeguard: The descent lemma supplies a sufficient step-size bound for passing the Armijo test on smooth fixed-batch trial segments.The experiment uses capped same-minibatch backtracking as a numerical safeguard rather than as a convergence result.

A. Data, model, and stress conditions

The study uses a frozen 60-writer FEMNIST setup with a compact convolutional classifier and evaluates generalization to new examples from known clients. Stress conditions alter labels for a fixed client group while preserving shared experimental comparisons.

  • Data: The primary study uses 60 writers as clients, with 8,064 training, 995 calibration, and 1,021 evaluation examples spanning all 62 character classes.Calibration and evaluation examples are disjoint from training but come from the same writers.
  • Model: The 27,326-parameter classifier has two convolutional layers, a 64-unit fully connected layer, and 62 output logits, evaluated by cross-entropy and accuracy.The regularizer is applied only during local training.
  • Optimization setting: The nonconvex objective is smooth within fixed activation and pooling patterns, but ReLU and max-pooling boundaries introduce nondifferentiabilities.The affine proposition describes prescribed directions rather than the complete nonlinear client trajectory.
  • Stress conditions: Each round samples 18 of 60 clients, while stress conditions keep 12 relabeled clients fixed for each seed across all 100 rounds.

B. Exponent protocol, pairing, and evaluation

The exponent study fixes the regularization reference pair and varies only consecutive integer powers under paired, budget-matched conditions. Evaluation combines predictive outcomes with client-displacement tail diagnostics and paired bootstrap intervals.

  • Exponent protocol: The study fixes (µ, R) = (0.03, 1) and compares consecutive integer exponents p = 2 through 8 to isolate exponent effects.Real-valued exponents remain theoretically admissible but are left for future investigation.
  • Stress protocol: The protocol caption specifies that stress multipliers affect only 12 clients selected once per seed and held fixed throughout training.
  • Paired comparison: All powers share initial networks, stress groups, participants, minibatch streams, and local-step, backward-pass, and communication budgets.Capped Armijo searches test at most 13 same-minibatch step sizes.
  • Evaluation: Primary evaluation uses pooled cross-entropy and accuracy on 1,021 examples, supplemented by the tenth percentile of writer accuracies.The displacement diagnostic is computed from selected-client norms each round.
  • Evaluation: The upper-tail-to-median statistic uses the 17th and 9th ordered norms among 18 selected clients, with larger values indicating a more dominant upper tail.
  • Statistical comparison: Paired bootstrap intervals enumerate all 3,125 ordered resamples of five seedwise loss differences, emphasizing effect sizes and paired directions for the exploratory study.

VI. PRIMARY FEMNIST RESULTS

The common-parameter FEMNIST study finds similar clean losses but improved stressed performance at intermediate exponents, with the best loss at p = 7 under moderate stress and p = 6 under severe stress.

  • 2.4095 at p = 7 is the lowest moderate-stress loss, compared with 2.7208 at p = 2.
  • 2.6235 at p = 6 is the lowest severe-stress loss, while losses rise to 2.6772 at p = 7 and 2.6701 at p = 8.
  • Clean losses remain close across exponents, ranging from 2.2610 to 2.2922.
  • The clean-plus-moderate calibration score is lowest at p = 7 with 2.3365, while p = 5 and p = 6 remain close at 2.3439 and 2.3429.
  • The results support an intermediate-power plateau rather than a uniquely optimal exponent.

B. Displacement compression and local cost

Increasing p compresses client-displacement disparities, but beyond an intermediate range predictive gains stop while Armijo trial-point costs continue increasing.

  • The severe-stress displacement-tail statistic falls from 4.936 at p = 2 to 2.086 at p = 8, a reduction of about 57.7%.
  • Increasing p narrows group separation by increasing unmodified-client movement initially and reducing stress-test-client movement, before unmodified movement turns downward at larger powers.
  • Severe-stress Armijo trial-point cost rises from 7835.0 at p = 2 to 9553.6 at p = 6, 9764.6 at p = 7, and 10069.6 at p = 8.
  • Beyond an intermediate range, additional displacement compression no longer improves prediction and continues to increase local search cost.

VII. DISCUSSION: CHOOSING THE EXPONENT

The exponent p should be calibrated to balance displacement suppression, preserved magnitude differences, and local search cost rather than maximized. In this FEMNIST protocol, p=5–7 provides the most useful range, while larger powers increasingly compress displacements and raise curvature-related costs.

  • VII. DISCUSSION: CHOOSING THE EXPONENT: p=5–7 provides the most useful range in the FEMNIST protocol, with different exponents favored for moderate-stress loss, accuracy, severe-stress robustness, and local cost.p=5 is the conservative low-cost choice, p=6 favors severe-stress robustness, and p=7 minimizes moderate-stress loss.
  • VII. DISCUSSION: CHOOSING THE EXPONENT: p=8 produces the smallest displacement tail but does not improve predictive performance and requires the most trial evaluations.This rules out the conclusion that increasing the exponent is always better.
  • VII. DISCUSSION: CHOOSING THE EXPONENT: The exponent should be selected jointly with µ and R by validation, but the fixed-parameter comparison does not establish the jointly optimal triplet (p, µ, R).A stronger tuning study should use several calibration seeds while preserving the evaluation seeds.
  • VII. DISCUSSION: CHOOSING THE EXPONENT: The study is limited to one 60-writer subset, one architecture, five seeds, and a composite stress intervention that changes multiple factors together.It does not test unseen writers, isolate individual stress components, or explore fractional powers.
  • VII. DISCUSSION: CHOOSING THE EXPONENT: Larger powers compress relative displacement disparities and approach fixed-radius behavior, while increasing local curvature outside the crossover region.The affine response explains the compression, and the curvature analysis predicts greater local search difficulty.
Loading 2609.09904v1…