PaperlayerView exact revision on arXiv

Source-linked AI summary

RELIEF: Turning Missing Modalities into Training Acceleration for Federated Learning on Heterogeneous IoT Edge

Beining Wu, Zihao Ding, Jun Huang

arXiv:2604.04243v1cs.NI

TL;DR

Heterogeneous IoT FL couples device capability and modality availability, causing stragglers to provide incomplete updates while wasting computation on absent-sensor parameters. RELIEF partitions the fusion-layer LoRA matrix into modality-aligned blocks, aggregates each within its modality cohort, and allocates training budgets by cohort divergence. Across PAMAP2 and MHEALTH under CNN and LoRA training, it reports up to 9.41× speedup, 37% energy reduction, and 15.3 pp rare-modality F1 gains over FedAvg.

  • Problem

    Coupled system-modality-data heterogeneity makes low-capability stragglers produce incomplete gradients, while standard aggregation dilutes rare-modality information and trains absent-sensor parameters.

  • Method

    RELIEF partitions the fusion-layer LoRA projection into modality-aligned column blocks used for cohort-wise aggregation and divergence-guided modality-aware elastic training.

  • Results

    Up to 9.41× speedup, 37% energy reduction, and 15.3 pp rare-modality F1 gains over FedAvg are reported across two IoT sensor datasets and two training backbones.

  • Takeaways & Limitations

    Cohort-wise aggregation removes cross-modal interference from the convergence bound, while divergence-guided allocation achieves sublinear regret.

Abstract

from arXiv · show

Federated learning (FL) over heterogeneous IoT edge devices faces coupled system-modality-data heterogeneity: the lower-cost device carries both fewer sensors and less computational power, so the slowest device (straggler) produces the most incomplete gradient signals. Naively averaging their updates dilutes rare-modality information and wastes computation on absent-sensor parameters, whereas existing methods handle the triple heterogeneity (system, modality, data) in isolation and none addresses their coupling. To resolve this issue, we propose RELIEF, a framework that partitions the fusion-layer Low-Rank Adaptation (LoRA) projection matrix into modality-aligned column blocks and uses this partition as a unified interface for aggregation, elastic training, and communication. Each block is aggregated only within the cohort of devices possessing that modality, which eliminates cross-modal gradient interference; the server then allocates personalized training budgets by prioritizing blocks with the highest cohort-internal divergence, so that resource-constrained devices train fewer but more impactful parameters. We prove that cohort-wise aggregation removes interference from the convergence bound and that the divergence-guided allocation achieves sublinear regret. Experiments on two IoT sensor datasets (PAMAP2, MHEALTH) under both full-parameter (CNN) and parameter-efficient (LoRA) training show that RELIEF achieves up to 9.41x speedup and 37% energy reduction over FedAvg with up to 15.3 pp rare-modality F1 gains, and real-device validation on a two-Jetson AGX Orin testbed confirms these results.

I. INTRODUCTION

Heterogeneous IoT edge FL couples device capability, modality availability, and data differences: the least capable devices are also the stragglers with the most incomplete updates. RELIEF addresses this coupling through modality-aligned parameter blocks shared across aggregation and elastic training.

  • Lower-cost IoT devices carry fewer sensors and less computational power, making the slowest devices produce the most incomplete gradient signals.
  • FedAvg averages structurally incompatible updates, diluting rare-modality signals and corrupting shared-modality representations.
  • Single-modal stragglers waste computation training parameter groups for modalities they do not possess, prolonging synchronous round times.
  • Existing multimodal FL methods improve aggregation but assume homogeneous device capabilities, while system-heterogeneous methods target single-modality models without modality semantics.
  • RELIEF partitions the fusion-layer LoRA matrix into modality-aligned column blocks for cohort-wise aggregation, divergence-guided elastic training, and on-demand communication.
  • The framework is evaluated on PAMAP2 and MHEALTH, with reported wall-clock speedup, maintained or improved accuracy, and rare-modality gains over baselines.

C. Low-Rank Adaptation in Federated Learning

LoRA represents the fusion-layer update with a low-rank residual whose projection matrix can be partitioned into modality-specific column blocks. RELIEF uses these blocks to align device updates, parameter groups, and communication with the modalities available on each device.

  • Low-Rank Adaptation: LoRA freezes pretrained weights and learns a low-rank residual ΔW = BA with rank ρ much smaller than the input and output dimensions.
  • Modality-Aligned Structure: Because the fusion input concatenates modality features, the LoRA projection A is partitioned into contiguous blocks A_m, each processing one modality.
  • Modality-Aligned Structure: A device lacking modality m receives no gradient for A_m, making modality-specific blocks natural units for modality-aware training and communication.
  • Parameter Groups: The model organizes trainable parameters into modality blocks, shared projection B, encoder parameters, and task-head parameters.
  • Federated Training: Each device can update only parameter groups corresponding to its modalities, uploads the groups selected by its elastic budget, and contributes to the next server aggregation.
  • Federated Training: Full-modality and single-modality devices produce gradients across different fusion blocks, so full-matrix FedAvg conflates structurally different updates.

IV. PROPOSED METHOD

RELIEF uses modality-aligned LoRA column blocks as a shared interface for diagnosing divergence, assigning training, and aggregating updates within modality cohorts.

  • Motivational studies: PAMAP2 diagnostics use eight clients across three device types and partition the LoRA projection matrix into four modality-aligned column blocks.The setup trains with standard FedAvg for 200 rounds.
  • Motivational studies: Acc-block update similarity falls to 0.41 for Full–Acc-only pairs versus 0.78 for Full pairs, indicating interference in a shared modality block.Absent Mag and HR blocks show near-zero similarity, while cross-modal interactions in full-device gradients corrupt shared Acc representations under FedAvg.
  • Motivational studies: Rare Mag and HR blocks show growing cohort-internal divergence, whereas the Acc block maintains low, stable divergence across training phases.The rare-modality cohorts contain three devices, compared with eight for Acc.
  • RELIEF framework: The same blocks serve as the aggregation boundary, elastic allocation unit, and communication granularity in RELIEF’s cyclic protocol.The protocol combines server divergence computation, device-specific training, and cohort-wise aggregation.
  • Modality-Decomposed LoRA: RELIEF decomposes the fusion-layer LoRA matrix into modality-aligned column blocks and aggregates each block only among devices possessing its modality.This cohort-wise rule is designed to eliminate cross-modal gradient interference while preserving fusion-layer sharing.

2) Divergence-Guided Modality-Aware Elastic Training:

RELIEF assigns each device a divergence-prioritized subset of accessible parameter groups under a device-specific budget, reducing straggler work while emphasizing disagreeing blocks.

  • Divergence estimation: RELIEF computes cohort-internal divergence for each parameter group and smooths it with an exponential moving average.The same procedure applies to fusion blocks, modality encoders, and the task head using their respective cohorts.
  • Personalized allocation: The server selects the accessible groups with highest smoothed divergence subject to |S_n| ≤ k_n and mandatory inclusion of every available fusion block.A device with fewer sensors therefore has fewer mandatory fusion blocks; the assignment is solved greedily after mandatory inclusion.
  • Elastic budget: Each device receives a budget k_n determined by computational capacity, communication overhead, per-group training time, and a target round time.The target T* is selected by binary search to minimize the maximum per-round time across devices.
  • Communication and energy: Devices train only assigned groups and upload only those groups, so a single-modality device sends |M_n|/M of a full-modality device’s volume.Reduced training and communication are accompanied by lower average power draw and compounded energy savings.

C. Training Pipeline

RELIEF’s training pipeline initializes divergence estimates, repeatedly allocates device-specific groups, performs restricted local updates, and aggregates each group within its active cohort.

  • Initialization: RELIEF begins with one full-training round, after which the server computes initial divergence estimates before entering the iterative protocol.The input includes device modality sets, a round count, an EMA coefficient, and a time target.
  • Server allocation: At each round, the server computes EMA-smoothed group divergence, determines each device’s budget, solves top-k allocation, and sends the model with assigned groups.The assignment is restricted to groups accessible under the device’s modality configuration.
  • Local training: Devices train assigned groups in parallel using full-model forward passes with zero-padded missing modalities, then upload their updates and modality configurations.Gradient computation and parameter updates remain restricted to the assigned groups.
  • Server aggregation: The server aggregates each fusion block and encoder within its active modality cohort, applies modality-count weighting to B, and averages H across devices.The updated divergence estimates then seed the next round.

V. THEORETICAL ANALYSIS

The theoretical analysis decomposes FedAvg error under explicit smoothness, variance, heterogeneity, and missing-modality assumptions, then shows cohort-wise aggregation removes scaling and cross-modal interference terms.

  • Convergence and allocation guarantees: The analysis also establishes a convergence rate and optimality and regret guarantees for divergence-guided elastic allocation.These results are presented through Theorem 3 and Propositions 4–5.
  • Assumptions: RELIEF’s guarantees rely on per-group smoothness, bounded stochastic variance, bounded heterogeneity, and near-zero missing-modality gradients.The fourth assumption models zero-padded-input noise with ε_0 → 0 and is supported by the diagnostic study.
  • FedAvg error decomposition: FedAvg’s fusion-block error decomposes into scaling bias, cross-modal interference, and irreducible intra-cohort disagreement.The variance component is averaged over local steps and cohort sizes.
  • Cohort-wise guarantee: Under RELIEF, the scaling and interference terms vanish because each block uses an exact active-cohort average without zero-padded gradients.The remaining error is the cohort-mean estimation disagreement.

C. Convergence Analysis

RELIEF’s convergence analysis decomposes updates across parameter groups and shows that cohort-wise aggregation removes cross-modal interference from the convergence bound. Elastic allocation further reduces the cohort residual by increasing cohort sizes for high-divergence modalities.

  • Theorem 3 establishes RELIEF’s convergence guarantee after R rounds under the stated assumptions and learning-rate schedule.The theorem uses learning rate η = O(1/√R) and E local epochs per round.
  • RELIEF’s convergence bound decomposes the global update across fusion blocks, encoders, and the task head, aggregating each within its respective cohort.The proof combines a smooth-descent argument with Theorem 2 for fusion blocks and standard FedAvg bounds for the remaining groups.
  • The cohort residual is the only aggregation-dependent term in RELIEF’s bound.Unlike RELIEF, FedAvg incurs additional cross-modal interference that does not vanish with more communication rounds.
  • Elastic allocation further reduces the cohort residual by enlarging min_r |C̃_m| for high-divergence modalities.The allocation targets modalities whose cohort-internal divergence is highest.

D. Elastic Allocation Analysis

RELIEF allocates the elastic training budget using modality-block divergence under a total budget constraint. The resulting greedy strategy preserves the optimal ranking and achieves sublinear cumulative regret rather than the linear regret of uniform or random allocation.

  • The elastic allocation minimizes a weighted cohort residual under the aggregate budget constraint Σ_m x_m ≤ K.Here, m denotes a modality block’s divergence and x_m denotes its cohort size.
  • The optimal allocation assigns resources proportionally to the square root of each block’s divergence.The divergence-guided greedy strategy selects groups in descending estimated-divergence order as a discrete approximation.
  • EMA-based divergence tracking bounds estimation bias when true divergence varies by at most δ between rounds.The analysis models temporal variation through δ = max_j,r |d_{r+1}^j − d_r^j|.
  • The divergence-guided allocation achieves O(√R) cumulative regret, whereas uniform or random allocation incurs O(R) regret.The sublinear rate follows from the stated follow-the-leader analysis.

A. Experimental Setup

The evaluation uses PAMAP2 and MHEALTH with realistic modality and compute configurations, comparing classical, elastic, multimodal, and federated-LoRA baselines under CNN and LoRA backbones. Macro-F1, wall-clock speedup, communication, and energy are measured across matched protocols.

  • Datasets: PAMAP2 and MHEALTH are multimodal human-activity-recognition datasets used to model realistic IoT sensor heterogeneity.PAMAP2 contains accelerometer, gyroscope, magnetometer, and heart-rate modalities; MHEALTH contains accelerometer, gyroscope, magnetometer, and ECG modalities.
  • Datasets: PAMAP2 uses 8 clients and MHEALTH uses 10 clients with full-, dual-, and single-modality device configurations.The configurations pair fewer modalities with slower devices, including 55× slower Acc-only clients.
  • Metrics: The study reports macro-F1, wall-clock speedup relative to FedAvg, and per-round communication volume.Both datasets use 5.12-second windows with a 1-second stride.
  • Baselines: Ten baselines span classical FL, system-heterogeneous or elastic FL, and multimodal or federated-LoRA methods.All methods use identical data splits, device configurations, and communication protocols.
  • Backbones: RELIEF is evaluated with a lightweight CNN and a pretrained Transformer plus LoRA, covering full-parameter and parameter-efficient training.The CNN partitions its fusion weight matrix into modality-aligned blocks, while the LoRA backbone partitions its fusion-layer projection matrix for MDLoRA.
  • Training configuration: The experimental setup uses 200 communication rounds, five local epochs, Adam optimization, and FLOP-proportional device profiling.Full, Acc+Gyro, and Acc-only clients are profiled at 275, 21, and 5 TOPS, respectively.
  • Energy measurement: The energy model combines active-training, communication, and idle-waiting power, with real-device monitoring reserved for later validation.The datasheet-based estimate is explicitly approximate.

B. Main Results

Across CNN and LoRA settings, RELIEF combines strong accuracy with faster training and improved rare-modality performance, while remaining robust to wider compute gaps and larger fleets. Ablations show that cohort-wise aggregation and divergence-guided allocation are central to this trade-off, with real-device deployment validating the approach on Jetson hardware.

  • Main results: Under Backbone 1, RELIEF reaches 93.7% F1 on MHEALTH with 2.87× speedup and 63% energy reduction versus FedAvg.On PAMAP2, it trades 1.9 percentage points for the same acceleration.
  • Main results: Under Backbone 2, RELIEF achieves 9.41× speedup, 37.5% communication savings, and competitive PAMAP2 F1 of 74.9% versus FedAvg’s 78.3%.On MHEALTH, RELIEF reaches 83.4% versus FedAvg’s 63.2%.
  • Convergence: RELIEF overtakes all baselines by round 40 in the reported B1 convergence settings and is the only method above 70% on both B2 datasets.Harmony collapses under B2 because it excludes the LoRA fusion layer from federation.
  • Ablation study: Removing cohort-wise aggregation lowers F1 by 6.4–7.2 pp under B1 and 8.1–9.6 pp under B2.The larger MHEALTH drop reflects its smaller ECG cohort.
  • Ablation study: Random allocation produces the largest ablation loss, reaching −7.0 to −8.2 pp under B1 and −9.6 to −11.3 pp under B2.Its budget and speedup match the no-cohort-aggregation variant.
  • Ablation study: Disabling elastic training raises F1 by 1.8–3.9 pp but reduces speedup to 1.66×/1.52× and increases energy from 312 J to 578 J under B1.The paper presents this as a latency–accuracy trade-off.
  • Robustness: RELIEF loses only 1.9 pp as the compute gap widens from 10× to 100× under B1 and remains advantageous as fleet size increases.Its MHEALTH advantage over FedAvg persists across the tested fleet sizes.

VII. REAL-DEVICE DEPLOYMENT

Real-device deployment on two Jetson AGX Orin clients validates RELIEF’s efficiency gains under physical edge constraints. Across CNN and LoRA backbones, RELIEF reduces round time and fleet energy while preserving competitive F1, though fixed forward costs limit LoRA speedup.

  • Deployment setup: The testbed uses two Jetson AGX Orin clients, with configurable power modes emulating full-modality high-compute and reduced-modality lower-compute devices.Orin-1 runs at 60 W as Type-A, while Orin-2 runs at 15 W or 30 W as Type-C or Type-B.
  • Timing results: Under Backbone 1, RELIEF reduces round time from 9.05 s to 4.70 s, achieving a 1.93× speedup over FedAvg.RELIEF lowers Type-C compute from 8.2 s to 3.4 s and shifts the bottleneck to Type-A.
  • Timing results: Under Backbone 2, RELIEF reduces round time from 19.02 s to 13.68 s, achieving a 1.39× speedup because the fixed forward pass dominates execution.The frozen MOMENT encoder imposes an approximately 12 s forward cost, while reducible backward time falls from approximately 6 s to 1.5 s.
  • Energy results: 37% fleet-energy savings are achieved under Backbone 1, with per-round energy falling from 1346 J for FedAvg to 846 J for RELIEF.Device-level energy is 48 J for RELIEF versus 118 J for FedAvg under Backbone 1, and 180 J versus 259 J under Backbone 2.
  • Accuracy and validation: Under Backbone 1, RELIEF reaches 89.6% F1 at 169 kJ, while FedAvg reaches 91.1% F1 at 269 kJ; under Backbone 2, RELIEF reaches 82.8% at 520 kJ versus FedAvg’s 63.1% at 690 kJ.Real-device F1 differs from simulation by less than 1 pp for RELIEF, while the overall LoRA speedup is lower than simulation because FLOP-proportional timing omits fixed forward costs.
Loading 2604.04243v1…