PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Uniform Compression: Budgeted Transmission Allocation for Extreme Federated Learning

Pengfei Li, Mohammad Khalil

arXiv:2609.39646v1cs.LG

TL;DR

Under extremely small per-round upload budgets, existing methods often allocate compression uniformly despite parameter blocks having unequal communication value. LBAT allocates bytes adaptively across layers and outperforms compression baselines in tabular prediction and generation under extreme budgets.

  • Problem

    Under extremely small per-round upload budgets, existing methods often allocate compression uniformly despite parameter blocks having unequal communication value.

    Source: Introduction · p. 1

  • Method

    LBAT estimates layer transmission value from local training signals and uses an exact-byte dynamic-programming allocator to choose layer configurations under strict budgets.

    Source: Introduction · p. 2, Solving the Allocation via Dynamic… · p. 4

  • Results

    Across federated tabular prediction and generation, LBAT outperforms uniform-rank, uniform-quantisation, FLoCoRA-style, and sensitivity-free baselines, improving communication–utility trade-offs under extreme budgets.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    LBAT is useful when parameter-block values are heterogeneous and complements quantisation, low-rank approximation, and temporal aggregation as candidate allocation actions.

    Source: Discussion · p. 7

    The evaluation is limited to tabular benchmarks; extending LBAT to sequential vision or foundation models remains unexplored.

    Source: Discussion · p. 7

Abstract

from arXiv · show

Federated learning faces severe communication bottlenecks when clients upload high-dimensional model updates. Existing methods often compress these updates uniformly across all layers. This uniform approach ignores the heterogeneous value of different parameter blocks and wastes limited bandwidth on insensitive layers. To address this issue, we propose Layer-wise Budgeted Adaptive Transmission (LBAT). LBAT reframes federated communication under extreme uplink budgets as a resource allocation problem. Our framework dynamically estimates the transmission value of different layers utilising local training signals. It then employs an exact byte dynamic programming allocator to determine optimal rank and bit configurations under strict budgets. We validate LBAT on highly heterogeneous federated tabular prediction and data generation tasks. Extensive experiments demonstrate that LBAT consistently outperforms uniform rank, uniform quantisation, and fixed compression baselines across various extreme budget regimes. Furthermore, it achieves significantly better communication and utility tradeoffs while preserving essential distributional fidelity.

Introduction · Related Work

LBAT reframes federated communication under extreme uplink budgets as layer-wise value allocation, estimating layer transmission value from local training signals and adapting upload configurations to strict byte budgets. In federated tabular prediction and generation, it outperforms uniform and fixed-compression baselines in communication-utility trade-offs.

Introduction

  • FedAvg enables local training with periodic server aggregation, but repeated uploads of high-dimensional updates remain a bottleneck as client numbers, model scales, or uplink constraints increase.Communication-efficient methods have addressed this through compression, synchronization-frequency control, and partial client participation.Source: Introduction · p. 1

Related Work

  • Prior approaches include stochastic quantisation in QSGD (Alistarh et al. 2017), gradient sparsification in Deep Gradient Compression (Lin et al. 2017), and periodic averaging, partial participation, and quantised messages in FedPAQ (Reisizadeh et al. 2020).LoRA reduces trainable parameters with low-rank adapters (Hu et al. 2022), and federated variants upload or quantise lightweight adapters (Babakniya et al. 2023).Source: Introduction · p. 1
  • Uniform quantisation, rank allocation, Top k sparsification, and fixed LoRA configurations assume parameter blocks have similar communication value, although layers differ in scale, compression sensitivity, and task utility.Client data heterogeneity and aggregation shift further vary layer-wise model differences across clients.Source: Introduction · p. 1

Introduction

  • LBAT reframes extreme-budget communication from uniform compression to preferential allocation of upload bytes, dynamically estimating layer transmission values from local training signals and adapting configurations under strict budgets.Its pipeline is illustrated in Figure 1.Source: Introduction · p. 2
  • Across federated tabular prediction and generation, LBAT outperforms uniform-rank, uniform-quantisation, FLoCoRA-style compression, and sensitivity-free allocation baselines in communication-utility trade-offs under extreme budgets.The scenarios include highly non-IID data, mixed features, and extreme sample skew.Source: Introduction · p. 2
  • LBAT is a lightweight framework that needs no public data, extra generators, or server-side distillation, using local value estimation and adaptive allocation to maximise transmission marginal utility under strict budgets.The paper frames this as shifting from how much to compress toward what to transmit preferentially.Source: Introduction · p. 2
  • The paper’s diagnostics examine layer-wise allocation, gap-to-full-upload, TRTR-normalized utility, and class coverage to investigate LBAT’s performance gains and interpretability.The empirical validation covers FedLBAT-Pred and FedLBAT-Syn.Source: Introduction · p. 2

Communication Efficient Federated Learning

Federated uploads remain bandwidth-limited despite communication-round reductions, prompting methods that compress client updates. This paper frames extreme-budget uploads as deciding which layers or parameter blocks merit transmission.

  • FedAvg reduces communication rounds through local multi-step updates and server-side averaging, but clients still upload model updates each round and uplink bandwidth remains a primary limitation.The passage highlights cross-institution collaboration and privacy-sensitive tabular data scenarios (McMahan et al. 2017).Source: Communication Efficient Federated… · p. 2
  • QSGD, Deep Gradient Compression, and FedPAQ reduce communication costs by quantising or sparsifying updates, focusing on how to compress given updates.The passage cites QSGD (Alis- tarh et al. 2017), Deep Gradient Compression (Lin et al. 2017), and FedPAQ (Reisizadeh et al. 2020).Source: Communication Efficient Federated… · p. 2
  • L GreCo uses layer-wise compression sensitivity to select compression parameters under error constraints, whereas this paper allocates transmission across layers or parameter blocks under extreme uplink budgets.L GreCo is attributed to Markov et al. 2024.Source: Communication Efficient Federated… · p. 2

Parameter Efficient Updates in Federated · Learning

Parameter-efficient updates reduce federated communication by sending lightweight low-rank adapters, while this paper treats those adapters as transmission units to allocate under single-round uplink budgets.

Learning

  • LoRA freezes the main model and adds trainable low-rank matrices, reducing the number of trainable parameters (Hu et al. 2022).Source: Learning · p. 2
  • FLoCoRA combines federated LoRA training with quantisation to further reduce communication costs.Source: Learning · p. 2
  • HetLoRA lets clients use different LoRA ranks and provides corresponding aggregation strategies (Cho et al. 2024).Source: Learning · p. 2
  • Unlike methods that treat adapter structures or rank as model-design choices, this paper makes low-rank adapters selectable and encodable transmission units.Source: Learning · p. 2
  • The proposed approach jointly allocates adapter rank and bit width across layers for single-round uplink budgets.Source: Learning · p. 2
  • It also allocates which layer updates are uploaded, making upload decisions part of the budgeted transmission allocation.Source: Learning · p. 2

Federated Tabular Learning and Tabular · Modeling

Prior work applies diffusion models to mixed-type tabular generation, including federated and differentially private settings; this paper instead addresses transmission-resource allocation across layers or parameter blocks to improve utility retention under communication constraints.

Modeling

  • TabDDPM shows that diffusion models handle continuous and discrete mixed-type tabular data, outperforming several GAN/VAE alternatives across multiple benchmarks (Kotelnikov et al. 2023).Source: Modeling · p. 2
  • FedTabDiff enables collaborative mixed-type tabular diffusion-model training across client data owners without centralizing raw data (Sattarov, Schreyer, and Borth 2024).Source: Modeling · p. 2
  • DP FedTabDiff combines differential privacy with federated tabular diffusion (Sattarov, Schreyer, and Borth 2025).Source: Modeling · p. 2
  • Unlike these works on tabular generative-model construction and training, this paper focuses on allocating transmission resources across layers or parameter blocks given byte budgets.Source: Modeling · pp. 2–3
  • LBAT evaluates layer-wise sensitivity locally and uses an exact-byte dynamic-programming allocator to choose rank and bit configurations under a strict communication budget.Figure 1 overviews the framework.Source: Figure 1 · p. 3, Modeling · p. 3
  • LBAT allocates byte budgets across layers or parameter blocks to improve utility retention under communication-constrained conditions.Source: Modeling · p. 3

Methodology

LBAT formulates client uploads as layer-wise budget allocation, choosing transmission configurations using local sensitivities and byte costs. It uses exact-byte dynamic programming to allocate the budget and sample-size-weighted server aggregation without distillation or proxy data.

Methodology

  • LBAT selects configurations to minimize estimated losses within byte budgets using local parameter sensitivities and communication costs, then aggregates received updates by sample size.The server requires no distillation or extra proxy data.Source: Methodology · p. 3

Budgeted Federated Transmission

  • Each block receives candidate configurations, including retained-rank and quantisation-bit choices, with byte costs and an option to upload nothing.Any auxiliary parameters outside LBAT decisions are synchronised as ordinary federated parameters and deducted from the available budget.Source: Budgeted Federated Transmission · p. 3

Sensitivity-Aware Objective

  • LBAT weights block compression distortion by online sensitivity scores estimated from gradient statistics reused during local training.The weighting is motivated by block-wise perturbation analysis; near a local minimum or with error approximately orthogonal to the gradient, the utility drop is heuristically bounded by a second-order term.Source: Sensitivity-Aware Objective · p. 4

Solving the Allocation via Dynamic Programming

  • LBAT casts each block’s choice among discrete transmission configurations as a Multiple-Choice Knapsack Problem and solves it with exact-byte dynamic programming.The allocator minimizes weighted distortion within the byte budget and recovers layer-wise configurations by backtracking.Source: Solving the Allocation via Dynamic… · p. 4

Task Instances

  • LBAT is applied to federated tabular diffusion generation and binary classification, with transmission selection applied to model updates after local training.For generation, LBAT leaves the task’s generative loss unchanged; for prediction, it applies the same budgeted selection to predictive-model updates.Source: Task Instances · p. 4

Experimental Setup

LBAT is evaluated on heterogeneous federated tabular benchmarks in synthetic-data and real-prediction settings, across constrained upload budgets and against compression baselines and oracle references.

Datasets and Federated Scenarios

  • The experiments use Adult, Credit Default, and Cardio, partitioned across 20 clients with label and quantity skew.Each dataset reserves 20% for testing; the remaining data are partitioned with a Dirichlet label skew of α = 0.3, and four clients are sampled per communication round.Source: Datasets and Federated Scenarios · pp. 4–5
  • The primary Syn setting trains a federated TabDDPM-based diffusion model and evaluates generated tables with TSTR, while auxiliary Real applies the policy to a tabular MLP predictor.The synthetic model draws on TabDDPM and FedTabDiff (Kotelnikov et al. 2023; Sattarov, Schreyer, and Borth 2024).Source: Datasets and Federated Scenarios · p. 5

Baselines and Evaluation Protocol

  • Experiments span 0.05×–1.00× upload budgets and compare FedLBAT with five baselines, plus centralized and TRTR oracles.The 1.00× budget transmits all adapter blocks at maximum rank and highest precision alongside synchronised auxiliary parameters; baselines are Uniform-Rank, Uniform-Quant, FLoCORA-style (Hu et al. 2022), FedLAMA (Lee, Zhang, and Avestimehr 2023), and FedAvg-Full (McMahan et al. 2017).Source: Baselines and Evaluation Protocol · p. 5
  • The synthetic track reports TRTR-normalised utility and distributional metrics, while real prediction reports test AUC and F1.Synthetic-track measures also include marginal Wasserstein distance, correlation error, label gap, and categorical support; all main results are averaged over three random seeds.Source: Baselines and Evaluation Protocol · p. 5

Results

Results assess FedLBAT’s communication–utility frontier, consistency across budgets and datasets, learned allocation policy, distribution fidelity, and transfer to real tabular prediction. Synthetic-track utility is TRTR-normalized for cross-dataset comparability.

Communication and Utility Tradeoffs Under Extreme Budgets

  • FedLBAT forms a stronger communication–utility frontier than uniform and FLoCORA-style baselines across Adult, Credit Default, and Cardio, especially at 0.05×–0.25× budgets.The results attribute the advantage to layer-wise allocation preserving task-relevant information where uniform rank or bit-width assignments waste bandwidth.Source: Figure 2 · p. 5, Communication and Utility Tradeoffs… · p. 5

Consistency Across Budgets and Datasets

Interpretability of Adaptive Budget Allocation

  • LBAT allocates budget non-uniformly across blocks, increases selected ranks and bit widths as budgets grow, and shows a sensitivity–share correlation of r = 0.52.The positive correlation supports the interpretation that its local sensitivity proxy aligns with transmission decisions and that allocation is not another uniform compression rule.Source: Interpretability of Adaptive Budget… · pp. 5–6, Figure 4 · p. 6

Data Quality and Distribution Fidelity Diagnostics

  • FedLBAT achieves the highest average normalized utility while maintaining competitive Wasserstein distance, correlation error, label gap, and full categorical support.FLoCORA-style compression has weaker utility, higher Wasserstein distance, and lower categorical support, while FedLBAT preserves important tabular distributional properties.Source: Data Quality and Distribution Fidelity… · p. 6

Generalization to Real Tabular Prediction

  • On the synthetic TSTR track, FedLBAT’s AUC gains over the strongest non-FedLBAT peer are +0.0778, +0.0489, and +0.0534 at low, medium, and full budgets; on real prediction, it is essentially tied.The passage attributes the larger gains in tabular modeling to greater heterogeneity in layer values and the likelihood that uniform compression wastes budget.Source: Generalization to Real Tabular… · p. 6
  • Removing adaptive budget allocation or gradient-based layer sensitivity, or using random allocation, causes significant AUC drops across datasets under extreme compression.The ablation results support both components of FedLBAT’s value-guided dynamic programming design.Source: Generalization to Real Tabular… · p. 6

Discussion

Under extreme budgets, LBAT’s central contribution is allocating scarce bytes to parameter blocks whose errors matter more, rather than compressing all updates uniformly. The discussion finds this allocation most consequential for synthetic tabular modelling, while noting limits in scope, valuation guarantees, and privacy.

Discussion

  • Unlike uniform-rank or uniform-quantisation schemes and fixed low-rank compression, FedLAMA adapts aggregation timing but not within-round rank and precision assignment.These methods differ in how they handle block-level transmission choices under the scarce-byte allocation problem.Source: Discussion · p. 6
  • On real prediction, FedLBAT is largely tied with strongest peers; on synthetic modelling, uniform baselines retain fidelity but lower utility, while FLoCORA-style compression shows label and coverage failures.The discussion attributes the synthetic task’s greater difficulty to preserving label structure, mixed-type dependencies, fidelity, and downstream predictability together.Source: Table 2 · p. 7, Discussion · p. 7
  • LBAT compares candidate costs and distortions, directing capacity toward blocks whose errors matter more; sensitivity’s moderate alignment with allocated share supports—but does not prove—this mechanism.The framework treats quantisation, low-rank approximation, and temporal aggregation as candidate actions in a per-round allocation problem.Source: Discussion · p. 7
  • LBAT remains untested beyond tabular benchmarks, uses sensitivity as a proxy rather than a formal utility guarantee, and lacks formal privacy guarantees.The discussion identifies stronger valuation methods, differential privacy, and client fairness as future directions.Source: Discussion · p. 7

Conclusion

LBAT reframes federated communication under extreme upload budgets as value-based allocation across parameter blocks rather than uniform compression. It uses local sensitivity estimates and dynamic programming to prioritize bandwidth for layers that dominate downstream utility, improving utility while maintaining high data quality.

Conclusion

  • LBAT shifts the central challenge from compressing updates uniformly to distributing limited communication resources across parameter blocks.It frames federated communication as a budgeted value-allocation problem.Source: Conclusion · p. 7
  • LBAT uses local sensitivity estimates and a dynamic programming allocator to prioritize bandwidth for layers that dominate downstream utility.Source: Conclusion · p. 7
  • LBAT's empirical evaluation shows that adaptive allocation improves utility while maintaining high data quality.Source: Conclusion · p. 7
Loading 2609.39646v1…