A Measured Communication-Quality Frontier for Federated LoRA Fine-Tuning with Adaptive Phase-Switching

1 Introduction

• Adapter transmissions in every round can make cumulative communication decisive where uplink bandwidth is scarce or metered.Federated learning trains across clients without sharing raw data, while LoRA reduces trainable parameters.Source: §1 · p. 1
• Prior federated LoRA comparisons mostly report parameter-count ratios, and the paper identifies the absence of a shared measured-communication axis and transition-round accounting.FedIT aggregates A and B, FFA-LoRA freezes A at initialization, and FLoRA uses SVD while retaining full bidirectional cost.Source: §1 · pp. 1–2
• The five-protocol frontier has a knee at ReverseAdaptive, beyond which additional savings cost roughly five times more quality per point.Quality uses held-out instruction-following loss; the LLaMA-3.2-3B comparison instead reports final training loss and downstream accuracy.Source: §1 · p. 2
• The byte-tracked B-only protocols account explicitly for the transition round, creating a one-round asymmetry in when upload and download savings begin.Internal checks match FLoRA's measured total to the byte and reproduce a matched FLoRA run to 10 decimal places when the switch is disabled.Source: §1 · p. 2
• ReverseAdaptive saves 40.5% communication at TinyLlama-1.1B and 30.0±4.0% at LLaMA-3.2-3B over FLoRA; at the latter scale, the same τ switches at rounds 7–9 without retuning.FFA-LoRA saves 61.9% at TinyLlama scale for 0.0182 more held-out loss than ReverseAdaptive.Source: §1 · p. 2

2.1 LoRA and Parameter-Efficient Fine-Tuning

• LoRA reduces trainable parameters by representing weight updates as low-rank products, making it suitable for adapting large pretrained models under storage and communication constraints.The paper describes the updates as BA, with B of size d × r and A of size r × k, where r is much smaller than min(d, k).Source: §2.1 · p. 2

2.2 Federated LoRA Aggregation

• FedIT averages client A and B matrices, FFA-LoRA freezes A at initialization, and FLoRA stacks client modules and applies truncated SVD; the proposed B-only protocol learns A before freezing it.FFA-LoRA's theoretical upload saving is 50%; FLoRA transmits full A+B payloads in both directions.Source: §2.2 · pp. 2–3

2.3 Communication-Efficient Federated Learning

• Gradient compression and quantization methods reduce transmitted payloads, and the paper describes them as orthogonal to its B-only protocol.The cited approaches include QSGD, deep gradient compression, and FedPAQ.Source: §2.3 · p. 3

2.4 Adaptive Aggregation and Scheduling

• Prior adaptive federated methods modify local objectives, normalize updates, or adapt server optimization, but the paper identifies no prior federated LoRA work switching aggregation modes on a loss-plateau signal.The paper presents ReverseAdaptive as addressing that gap.Source: §2.4 · p. 3

3.1 Background and Notation

• Round communication cost is upload plus download bytes, and GQA makes the B-only share 36% of full state for TinyLlama-1.1B and 40% for LLaMA-3.2-3B.The method models LoRA updates as ΔW = BA, with clients training local adapters and uploading a representation each round.Source: §3.1 · p. 3

3.2 Bidirectional B-Only Protocol

• After K full-state FLoRA rounds, the transition round still uploads full state but broadcasts only B; subsequent rounds upload and broadcast B only.At transition, the server caches A from the first client's upload; the accounting yields download savings one round before upload savings.Source: §3.2 · pp. 3–4

3.3 ReverseAdaptive Aggregator

• After warmup, ReverseAdaptive switches permanently from FLoRA to FFA-LoRA when relative per-round loss improvement falls below dimensionless τ, with a stability reversion if loss rises over 10%.No reversion occurred in the reported runs; the loss-plateau signal needs no additional instrumentation, while gradient-norm and singular-spectrum alternatives are left for future work.Source: §3.3 · p. 4

3.4 Implementation Details

• Transmitted parameter bytes use float32 for TinyLlama and float16 for LLaMA-3.2-3B; the B-only fraction depends on GQA and target projections.For TinyLlama, the reported 36% applies to q_proj and v_proj targets and also to the four-projection attention set, but not to q_proj alone or sets including MLP projections. Non-IID skew uses instruction-length buckets rather than semantic labels.Source: §3.4 · pp. 4–5
• ReverseAdaptive's transition A is seeded from the first client's upload after local training, and the effect of its mismatch with the last client's locally adapted A on reported quality is unmeasured.Client ordering is fixed within each run, and averaging A across clients at transition is left to future work.Source: §3.4 · pp. 4–5

4.1 Setup

• The CUDA corpus supports the frontier, Dolly replication, and statistical tests, while MPS supports threshold ablation and 3B results; baselines were reimplemented in a shared training loop.Loss comparisons are made within a backend because cross-backend differences exceed some reported between-method differences.Source: §4.1 · pp. 5–6

4.2 Held-Out Instruction-Following Metric

• Held-out instruction-following quality is the mean token-level cross-entropy change from the unadapted base on a disjoint 500-example slice; more negative deltas are better.Held-out deltas are comparable within a dataset, not across datasets, because base-model losses differ.Source: §4.2 · p. 6

4.3 The Communication-Quality Frontier

• 40.53% savings at 1533.13 MB places ReverseAdaptive between FLoRA/FedIT (2578.13 MB) and FFA-LoRA (983.13 MB); it beats FFA-LoRA by 0.0182 held-out loss.Across the four distinct operating points, communication decreases as both final training loss and held-out instruction-following loss worsen.Source: §4.3 · p. 7, Table 2 · p. 8

4.4 Replication on Dolly-15k

• ReverseAdaptive’s held-out cost is 0.0063 on Alpaca and 0.0061 on Dolly, whereas Two-Phase K=8’s cost shifts 25.5% between datasets.The supported stability claim applies to ReverseAdaptive’s cost, not to the protocol family generally.Source: §4.4 · p. 8

4.5 Convergence and Switching Behavior

• ReverseAdaptive switches at round 6 and produces the steepest savings trajectory; phase transitions cause slight slope changes without destabilizing training.The final loss gap accumulates gradually after the switch rather than appearing as a sharp discontinuity.Source: §4.5 · p. 8, Figure 2 · p. 9

4.6 Downstream Evaluation

• At 1.1B, methods cluster within 1.0 percentage point and modestly regress; at 3B, they differ by at most 0.5 points, matching or exceeding base on three benchmarks but trailing on HellaSwag.At 3B, the HellaSwag shortfall is 0.1 to 0.5 percentage points.Source: §4.6 · pp. 8–9

4.7 Threshold Ablation

• τ≥0.005 saturates at a round-6 switch, while τ=0.001 and τ=0.002 switch later; thresholds from 0.005 to 0.2 produce identical results.The paper reports that the maximum-savings setting can be chosen anywhere in [0.005, 0.2] without precision loss.Source: §4.7 · p. 9

4.8 Scale Validation

• At LLaMA-3.2-3B, τ=0.01 saves 30.0 ± 4.0% without scale-specific tuning, versus 26.0% for Two-Phase K=8; ReverseAdaptive switches across rounds 7–9.The same threshold used at TinyLlama-1.1B produced this behavior without scale-specific tuning.Source: §4.8 · pp. 9–10

5.2 Choosing an Operating Point

• With τ=0.01, ReverseAdaptive switches at round 6 on TinyLlama and mean round 8.0±1.0 on LLaMA-3.2-3B without scale-specific retuning, but two scales and datasets do not establish a universal default.The relative-loss threshold is scale-free; the LLaMA-3.2-3B runs switched at rounds 7, 8, and 9.Source: §5.2 · p. 12
• 40.5 percentage points of communication savings from FLoRA to ReverseAdaptive cost 0.0063 in held-out loss; the next 21.3 points to FFA-LoRA cost 0.0182, 5.5 times more per point saved.The frontier’s knee is at ReverseAdaptive, where the first tranche of savings is available without choosing fixed K in advance.Source: §5.2 · p. 12

5.3 Measured Bytes and Theoretical Parameter Counts

• B-only transmission saves exactly 36% on TinyLlama-1.1B and 40% on LLaMA-3.2-3B, rather than the theoretical 50% suggested without accounting for their different GQA configurations.Source: §5.3 · p. 12
• The FLoRA-to-FFA-LoRA transition adds 55.0 MB on TinyLlama because upload remains full while broadcast is already B-only.Upgrading both directions would cost 110.0 MB; the look-ahead broadcast halves that transition cost.Source: §5.3 · p. 12
• FFA-LoRA’s measured 983.13 MB includes a 55.0 MB seeding upload; a design-faithful implementation reaches 928.13 MB and 64.00% savings, without changing the knee conclusion.For ReverseAdaptive, freezing the already-held global A instead is untested and would change the adapter as well as the bytes.Source: §5.3 · p. 13

5.4 Zero-Shot Benchmark Behavior and the Training Objective

• At TinyLlama scale, instruction tuning regresses on ARC-Easy, BoolQ, and HellaSwag, while held-out Alpaca loss falls from the base model’s 1.9352 to 1.3349–1.3608.At LLaMA-3.2-3B, the same procedure is flat-to-positive on MMLU, ARC-Easy, and BoolQ and within 0.5 percentage points of the base on HellaSwag; the held-out slice is same-distribution, not evidence of unrelated-task generalization.Source: §5.4 · p. 13

• Quality is held-out loss on same-distribution instruction examples; classification, longer training runs, generation metrics, and transfer to unrelated tasks are uncharacterized.Source: §6 · p. 14
• Experiments use Alpaca-3k and Dolly-15k, but FFA-LoRA and FedIT were evaluated only on Alpaca, so the cross-dataset comparison covers three protocols rather than the full five-method frontier.Source: §6 · pp. 13–14
• Results cover Llama-style decoder-only models with LoRA on q proj and v proj; other architectures, target sets, or rank configurations may yield different B-only fractions and savings.Source: §6 · p. 14
• The simulation assumes full client participation and tests length-bucket rather than semantic heterogeneity, leaving partial participation and semantic heterogeneity unstudied.Source: §6 · p. 14
• Real network latency, packet loss, and bandwidth are not modeled, so wall-clock communication time is not predicted; differential-privacy accounting is also absent.Source: §6 · p. 14
• At the switch, ReverseAdaptive freezes A from a client upload rather than the A clients trained B against; the effect on quality is unmeasured, and using the held global A is untested.Source: §6 · p. 14
• The LLaMA-3.2-3B seeds span three code revisions, combining seed and environment variation; evaluation noise is not separately characterized.TinyLlama downstream evaluation is single-seed per checkpoint despite three-seed training.Source: §6 · p. 14

7 Conclusion

• ReverseAdaptive locates the frontier knee without a preset boundary; savings beyond it cost roughly five times more held-out quality per point, while its advantage over FFA-LoRA is 0.0182 held-out loss at a cost of 550 MB.Learning A through a FLoRA phase before freezing it yields materially better adapters than freezing A at initialization.Source: §7 · pp. 14–15
• Byte-level tracking reveals a B-only fraction of 36% on TinyLlama-1.1B and 40% on LLaMA-3.2-3B, plus a 55.0 MB transition cost independent of switch timing.These quantities are not captured by parameter-count ratios.Source: §7 · p. 15
• An independent full-state implementation matches the measured total to the byte, and a closed form reproduces every full-participation switch-round total.Across backends, 27 of 31 runs with valid references agree within tolerance; every communication difference is exactly one step of the measurement lattice.Source: §7 · p. 15
• The same threshold transfers without retuning, firing at round 6 on TinyLlama-1.1B and a mean of round 8 on LLaMA-3.2-3B; transition quality cost is stable across the two tested datasets.Source: §7 · p. 15
• The conclusion calls reporting actual transmitted bytes, rather than parameter-count predictions, a prerequisite for protocol comparison as federated LLM deployment expands into privacy-sensitive domains.Source: §7 · p. 15

Statements and Declarations

• The author reports no funding and no competing interests; ethics approval, consent to participate, consent for publication, and materials availability are marked not applicable.Source: Statements and Declarations · p. 17
• Analysis CSVs, run records, regeneration scripts, and source code are publicly available, while adapter states are included only for three seed-42 TinyLlama-1.1B runs evaluated in Table 4.Source: Statements and Declarations · p. 17

Appendix A Experimental Details

• Experiments use AdamW with learning rate 10−4, β1=0.9, β2=0.999, weight decay 0.01, no learning-rate schedule, gradient clipping at 1.0, and a 256-token maximum sequence length.Source: Appendix A · p. 17

A.1 Reproducibility Checklist

• The release provides code and configuration files, 179-assertion claim verification, IID and partition-sensitivity seeds, and Alpaca and Dolly-15k matched subsets; adapter checkpoints are limited to three seed-42 TinyLlama-1.1B runs.A 34-run, approximately 285-hour MPS core was re-executed on rented NVIDIA hardware; Dolly-15k and FFA-LoRA and FedIT baseline runs used rented hardware only.Source: §A.1 · p. 17

A.2 Disabling the Switch

• With switching disabled, ReverseAdaptive reproduces the matched FLoRA [6] run to 10 decimal places at every round; τ=0 still triggers at round 11, so the appendix uses τ=−1.0 and warmup beyond the round budget to keep the wrapper inert.The observed loss rise means any non-improving round satisfies the switch criterion; the threshold must be below the most negative observed relative change, approximately −0.0009 here.Source: §A.2 · pp. 17–18

Appendix B Per-Seed Full Results

• The method comparisons use the CUDA corpus for TinyLlama-1.1B, MPS for LLaMA-3.2-3B, and the larger five-seed MPS set for α=0.1 partition sensitivity.Source: Appendix B · p. 18
• At α=0.5, mean final loss is statistically indistinguishable from IID for every method, with differences of 0.001 to 0.007 against per-seed standard deviations near 0.02 over three seeds.Source: Appendix B · p. 18
• Partition-draw sensitivity raises seed-to-seed standard deviation from 0.5 to 1.6 × 10−3 under IID to 18 to 25 × 10−3 at α=0.5 across methods.The per-seed spread at α=0.1 is also wide.Source: Appendix B · p. 18

• With three seeds, paired tests have very low power; Bonferroni’s 0.0083 threshold is not met by held-out-loss ReverseAdaptive vs. Two-Phase K=8 (p = 0.026).Source: Appendix C · p. 18
• p < 10−3 on both metrics: ReverseAdaptive versus FFA-LoRA remains the load-bearing comparison, with a difference roughly twenty times the largest per-method standard deviation.Source: Appendix C · p. 18
• The LLaMA-3.2-3B ReverseAdaptive vs. Two-Phase K=8 confidence interval [−0.0114, 0.0114] contains the 0.0043 TinyLlama difference, so the test cannot distinguish equivalence from that difference.Source: Appendix C · p. 18

• The primary corpus was produced on MPS and re-executed on CUDA; byte counts use numel×element size for tensors actually sent.Source: Appendix D · pp. 18, 20
• IID runs require per-round training loss agreement within 0.01; non-IID runs allow 0.025 but require an exact switch round and at most two consecutive rounds above 0.01.Source: Appendix D · p. 18
• At α=0.1, seeds 42 and 1000 had 9 and 8 active clients, scaling communication to 1379.81 and 1226.50 MB; cross-backend checks cover seeds 42, 123, and 456.Source: Table B2 · p. 19
• Held-out instruction-following loss was not computed for LLaMA-3.2-3B; quality at that scale is reported as final training loss and downstream zero-shot accuracy.Source: Table C1 · p. 20
• 27 of 31 comparable MPS–CUDA pairs agree within tolerance; three Two-Phase K=8 α=0.5 references are absent because those MPS runs predate bidirectional B-only and never switched.Source: Appendix D · p. 18, Table D1 · p. 20
• Three of four disagreements are one 110 MB switch-round step near tight-τ IID or α=0.1 boundaries; Run 16 fails only the consecutive-spike rule.Source: Appendix D · p. 18, Table D2 · p. 20
• A CUDA rerun of Run 17 again switched at round 7, while exact switch-round agreement held for IID τ=0.01 and all three LLaMA-3.2-3B ReverseAdaptive runs.Source: Appendix D · p. 20
• Across two CUDA revisions, FLoRA, Two-Phase K=8, and ReverseAdaptive byte totals matched at 2578.13, 1863.13, and 1533.13 MB; loss retains revision uncertainty.Source: Appendix D · p. 20