PaperlayerView exact revision on arXiv

Source-linked AI summary

Component-Aware Differential Privacy for Federated Multilingual Speech-LLMs

Jordi Luque, Fernando López, Aleix Sant

arXiv:2609.11762v1cs.CL

TL;DR

Speech-LLM federated training exposes a failure mode in single-pool per-layer DP when encoder and decoder update norms differ greatly. The paper introduces α-split, which separates component pools while preserving the original DP guarantee. At calibrated α, it restores utility near flat DP, provides 4.47× tighter encoder noise protection, and adds only 2.6% LLM noise overhead.

  • Problem

    Single-pool per-layer DP can suffer cross-component budget collapse in speech-LLMs with large encoder/LLM update-norm imbalances, harming WER or collapsing training.

  • Method

    α-split allocates independent clipping pools to the acoustic encoder and LLM, with budgets C√α and C√1−α, while preserving joint sensitivity and global DP guarantees.

  • Results

    α-Split-e stays within 0.003 absolute of Flat-e at WER 0.1959 on Whisper+TinyLlama, while providing 4.47× tighter encoder protection and only +2.6% LLM noise overhead.

  • Takeaways & Limitations

    Structural α-split is optimal for extreme component imbalances, while adaptive PFL-Dim+EMA-e is the more robust default for balanced architectures.

Abstract

from arXiv · show

Per-layer differential privacy (DP) clipping improves gradient fidelity in federated learning by allocating per-matrix clipping budgets proportional to parameter count. We show that this recipe breaks for speech large language models (speech-LLMs), when the acoustic encoder and the language decoder differ by an order of magnitude in update norm. Single-pool per-layer methods suffer \emph{cross-component budget collapse}, dragging word error rate (WER) far from flat global clipping or collapsing training entirely. When the norm imbalance is milder, adaptive single-pool methods partially recover, confirming that collapse severity scales with the inter-component norm ratio. We empirically diagnose the root cause across six per-layer methods and three speech-LLM architectures. We then propose \emph{$α$-split}, a two-pool allocation that normalises encoder and LLM parameters into independent pools, and show that joint $\ell_2$ sensitivity and the original $(\varepsilon,δ)$-DP guarantee are unchanged. At architecture-calibrated $α$, our method recovers WER utility compared to flat DP, while granting the encoder $4.47{\times}$ tighter per-component noise protection against speaker voice-based gradient-inversion attacks at only $+2.6\%$ LLM noise overhead.

I. INTRODUCTION

Speech-LLMs combine acoustic, connector, and language components, creating optimization asymmetries in federated, privacy-sensitive training. The paper diagnoses cross-component budget collapse in single-pool per-layer DP and proposes α-split component-aware allocation.

  • I. INTRODUCTION: Federated speech training protects privacy-critical audio but faces non-IID client data and poorly understood clipping effects in multimodal models.Clients differ across speakers, accents, microphones, and acoustic environments.
  • I. INTRODUCTION: Single-pool per-layer DP breaks under large encoder/LLM norm ratios because the encoder’s parameter count dilutes the LLM clipping budget.Across six methods, this produces severe WER degradation or complete training collapse.
  • I. INTRODUCTION: Adaptive single-pool clipping corrects mild encoder/LLM norm imbalances, whereas α-split separates encoder and decoder pools for extreme imbalance.The proposed split allocates budgets C√α and C√1−α while preserving joint sensitivity and global DP guarantees.
  • I. INTRODUCTION: Speech-LLMs couple an acoustic encoder, cross-modal connector, and language decoder to generate transcripts from speech and prompts.The encoder produces frame representations, the connector downsamples and projects them, and the decoder generates the transcript.

B. Central DP-FL Trust Model

The central trust model clips each client update before transmission and adds calibrated Gaussian noise during server aggregation. The paper treats the full parameter delta as a flat concatenation under FedAvg while studying heterogeneous trainable components.

  • B. Central DP-FL Trust Model: The trusted server aggregates client updates after client-side clipping and adds calibrated Gaussian noise to protect the released model sequence.The stated threat model is an external observer of the released model sequence.
  • B. Central DP-FL Trust Model: Speech-LLM updates include LoRA adapters and connector projections across an acoustic encoder, connector, and language decoder.The architecture maps speech through the encoder and connector before transcript generation by the decoder.
  • B. Central DP-FL Trust Model: The clipped aggregate has ℓ2 sensitivity C/n, and σ is the noise multiplier relative to that sensitivity.Experiments use C=1.0 and σ=0.1 with Rényi DP accounting.
  • B. Central DP-FL Trust Model: The encoder contains 73.1% of all LoRA parameters, although each encoder matrix is smaller than an LLM attention matrix.This parameter-count asymmetry motivates examining how clipping allocations interact with component structure.

C. Heterogeneous Dataset and Speaker Partitioning

Experiments use a multilingual audiobook corpus partitioned by speaker into highly heterogeneous federated clients. Gradient-norm measurements reveal separated encoder and LLM update scales, explaining cross-component budget collapse under flat clipping.

  • C. Heterogeneous Dataset and Speaker Partitioning: The MLS benchmark contains 685.7 hours of training data across eight European languages and 138 hours of held-out test data.The test split contains 19,492 samples.
  • C. Heterogeneous Dataset and Speaker Partitioning: Speaker-based partitioning assigns each speaker’s utterances to one of 316 clients, creating simultaneous linguistic and acoustic heterogeneity.Each client draws from at most one language and one acoustic identity, with microphone, room, and speaking-rate variation.
  • C. Heterogeneous Dataset and Speaker Partitioning: Federated training runs for 40 rounds with approximately 94 clients sampled per round and 10 local epochs before aggregation.The sampling rate is q=0.3, with AdamW optimization and maximum learning rate η=10^-4.
  • C. Heterogeneous Dataset and Speaker Partitioning: LLM matrices have approximately 0.1–0.2 update norms while encoder matrices are approximately 0.01–0.02, creating an order-of-magnitude gap.The encoder updates sit near the Gaussian noise floor σC/√n≈0.01, while LLM updates remain above it.
  • C. Heterogeneous Dataset and Speaker Partitioning: The approximately 10× per-matrix norm gap drives cross-component budget collapse because encoder matrices collectively absorb a disproportionate share of the C^2 budget.Under flat clipping, every LoRA matrix receives the same ratio scalar, while the LLM-dominated norm can suppress encoder scaling toward the noise floor.

B. SNR Suppression under Global Clipping

Global clipping gives both components the same clipping scale and noise floor, so dominant LLM and connector norms suppress encoder signal-to-noise. Single-pool alternatives allocate budgets by layer size or adapt them with EMA, but remain vulnerable to this imbalance.

  • SNR suppression: The encoder’s per-layer signal is approximately 10× weaker than the LLM’s, yielding SNRE ≈0.1 SNRL under global clipping.The shared noise floor and similar per-matrix parameter counts make the norm imbalance directly suppress encoder signal.
  • SNR suppression: Flat DP maintains a high global clip rate above 80%, leaving the encoder signal near the noise floor throughout training.LLM and connector norms dominate dynamically across the 40-round evaluation.
  • Evaluated methods: Per-layer baselines include flat, uniform, size-proportional, and EMA-adaptive allocations over LoRA matrices.The adaptive variants reweight budgets from observed gradient norms, while PFL-Dim uses size-proportional allocation.
  • Evaluated methods: α-Split treats the encoder and LLM+connector as independent pools with budgets C√α and C√1−α.The encoder-unfrozen suffix -e distinguishes experiments where the encoder is trainable.

A. The α-Split Design

α-Split assigns a tunable fraction of the squared clipping budget to the encoder and clips encoder and LLM+connector parameters with independent per-layer budgets.

  • A. The α-Split Design: α allocates C^2 between independent encoder and LLM+connector pools, whose per-layer budgets are computed separately.The encoder receives fraction α, while the LLM+connector receives the remaining fraction 1−α.

1) Formulation:

The connector is grouped with the LLM rather than the encoder, keeping each pool internally Euclidean-tight and preventing connector parameters from inflating the encoder budget.

  • 1) Formulation:: The LLM pool includes the connector because assigning it to the encoder would incorrectly inflate the encoder budget.For Whisper-based models, the connector’s approximately 4,096 parameters are negligible within the LLM pool.
  • 1) Formulation:: α-Split defines cE and cL as the clipped encoder and LLM norms before computing the joint ℓ2 sensitivity.The formulation combines the two independently clipped components for the sensitivity calculation.

2) Sensitivity and Privacy Preservation:

α-Split preserves the global sensitivity and DP accounting while redistributing effective noise across components. At α=0.05, it sharply tightens encoder protection with minimal LLM overhead.

  • 2) Sensitivity and Privacy Preservation:: At α*=0.05, the encoder receives 4.47× tighter per-component privacy while LLM noise increases by only 2.6%.Figure 4 compares α-Split-e’s component noise multipliers with the Flat-e reference.
  • 2) Sensitivity and Privacy Preservation:: The joint sensitivity remains C, so Gaussian noise, Rényi accounting, ε, and δ are unchanged from Flat-DP.This makes α a utility-versus-component-privacy parameter rather than a change to the global DP guarantee.
  • 2) Sensitivity and Privacy Preservation:: The selected α=0.05 retains 99.4% of the LLM’s Flat-e budget and gives the encoder a clip norm of 0.028 versus an observed delta-norm of 0.013.The selection criteria depend on parameter counts and the encoder-to-LLM update-norm ratio.
  • 2) Sensitivity and Privacy Preservation:: Flat-DP is treated as the utility ceiling at fixed (ε, δ), whereas α-Split’s advantage is asymmetric per-component privacy.The encoder’s effective noise multiplier is σ√α=0.447, while the LLM’s is σ√1−α=0.103.

5) Training Dynamics and Clipping Rate Analysis:

α-Split-e keeps encoder clipping low after warmup while the LLM pool remains more heavily clipped. Across frozen-encoder groups, PFL-Dim+EMA achieves the best DP WER, while α-Split offers no structural advantage.

  • Training dynamics: 51% encoder clipping in rounds 1–6 falls to 5–14% after stabilization, while LLM+connector clipping declines from 93% to 25–40% by round 40.The encoder norm settles below its 0.224 pool threshold, whereas the LLM+connector norm remains closer to its 0.975 threshold.
  • Encoder-frozen variants: PFL-Dim+EMA achieves the best frozen-encoder DP WER across W+TinyLlama, W+EuroLLM, and Voxtral, with WERs of 0.1657, 0.1362, and 0.1450.It outperforms Flat by 0.0028–0.0082 across the three groups.
  • Encoder-frozen variants: Equal-budget PFL-Uniform severely degrades frozen-encoder WER, reaching 0.4643 for W+TinyLlama and 0.3500 for W+EuroLLM.The result indicates that equal budget per layer is harmful even without cross-component coupling.
  • Encoder-frozen variants: α-Split slightly underperforms Flat for frozen EuroLLM, with gaps of +0.0128 versus +0.0114 because the encoder is inactive.When the encoder is frozen, all methods effectively allocate the budget to LLM layers, removing the split’s structural advantage.

B. Encoder-Unfrozen Variants and Cross-Component Collapse

Unfreezing the encoder exposes severe single-pool budget collapse under extreme norm imbalance, while α-Split preserves utility. Adaptive single-pool clipping is preferable when the imbalance is mild, so α must be architecture-calibrated.

  • Whisper+TinyLlama: α-Split-e stays within 0.003 absolute WER of Flat-e at 0.1959 on Whisper+TinyLlama, while Uniform variants collapse to 0.9944 and 0.8243.Additional seeds give std ≤0.02 for Flat-e, PFL-Dim+EMA-e, and α-Split-e, confirming stable rankings.
  • Whisper+EuroLLM: α-Split-e reduces the Whisper+EuroLLM gap to Flat-e from +0.0388 to +0.0143 at WER 0.1600, while PFL-Unif+EMA-e collapses to 0.4769.The result is reproduced on a stronger multilingual backbone, supporting an architectural source for the degradation.
  • Voxtral-Mini-3B: At a ≈1.7× Voxtral norm ratio, PFL-Dim+EMA-e achieves WER 0.1413, while α-Split-e reaches 0.1495 because α=0.05 over-clips the encoder.The paper contrasts this mild imbalance with ≈12× for Whisper+TinyLlama, where structural splitting is favored.
  • Operating regimes: The deployment boundary is ratio-dependent: α=0.05 is calibrated for ≥12× imbalance, whereas PFL-Dim+EMA is safer when the ratio is unknown or below 12×.The paper recommends selecting α from component delta-norms observed during non-DP warmup rather than applying it blindly.
  • Conclusion: Across the final comparison, α-Split-e is best for extreme imbalance, while PFL-Dim+EMA-e is best for balanced Voxtral; α-Split-e also improves Whisper+EuroLLM over the adaptive baseline by 2.45% absolute.These results establish distinct operating regimes rather than one universally optimal per-layer method.

APPENDIX A EUROLLM ENCODER LEARNING-RATE SENSITIVITY

Encoder learning-rate tuning removes the norm imbalance in Whisper+EuroLLM and reverses the relative ranking of adaptive single-pool clipping and α-Split. This separates optimization-scale effects from the DP mechanism itself.

  • Learning-rate sensitivity: Reducing the encoder learning rate to ×0.02 improves non-DP FedAvg WER to 0.1196 from 0.1757 under the default rate.The tuned result closely matches the frozen EuroLLM baseline of 0.1330.
  • DP results: At enc-lr ×0.02, PFL-Dim+EMA-e achieves WER 0.1440, outperforming α-Split-e at 0.1511 and Flat-e at 0.1525.The matched no-DP baseline is 0.1264, and the adaptive method improves from 0.1845 at the default learning rate.
  • Collapse diagnosis: At the default encoder learning rate, encoder norms are ∼10× larger than individual LLM-layer norms, allowing 64 encoder matrices to monopolize shared clip-budget slots.Reducing the learning rate by ×50 proportionally shrinks encoder norms and lets adaptive allocation concentrate budget on LLM layers.
  • Interpretation: Learning-rate tuning and α-split are complementary: tuning lowers the encoder signal-to-noise ratio under DP, whereas α-split preserves the encoder learning rate and provides 4× tighter protection.The controlled ablation attributes default-rate collapse to cross-component gradient-norm imbalance rather than a change in the DP mechanism.

B. Speaker Overlap Mechanics in audiobook Corpora

The audiobook-derived MLS partition creates speaker overlap across train and test splits despite speaker-based client assignment. The design preserves utterance disjointness while enabling both unseen-speaker and seen-speaker generalization evaluation.

  • Overlap mechanism: A 2.5% speaker overlap remains because LibriVox volunteers often record multiple books or chapters that standard ASR splits distribute across partitions.Eight of 316 clients, representing 2.8% of training samples, contain speakers also present in the test split.
  • Evaluation implications: The partition evaluates generalized performance on unseen speakers while also measuring adaptation to unseen utterances from speakers present in training.This follows from mapping each unique speaker to one client while retaining the benchmark split structure.
  • Data separation: No individual utterance is shared between training, validation, and test sets, preventing direct audio memorization or leakage.The overlap concerns speakers, not duplicated audio samples.
Loading 2609.11762v1…