Unveiling the Depth-Performance Dilemma in Split-Federated Fine-tuning of LLMs

Hariharan Ramesh; Someshwaran Murugaiyan; Jyotikrishna Dass

Source-linked AI summary

Unveiling the Depth-Performance Dilemma in Split-Federated Fine-tuning of LLMs

Hariharan Ramesh, Someshwaran Murugaiyan, Jyotikrishna Dass

arXiv:2608.22188v1Aug 23, 2026cs.LGcs.CLcs.DC

TL;DR

SFF must balance resource and privacy incentives for deep partitions against uncertain utility in LLM fine-tuning. The paper audits cut depths and adapter aggregation across models and benchmarks, finding consistent depth-related degradation and diagnosing structural causes in Transformer server partitions.

Problem
The paper examines whether deeper SFF partitions, favored for efficiency and privacy, preserve LLM fine-tuning utility and how aggregation behaves in this split setting.
Method
The paper performs exhaustive cut-layer evaluations across models, benchmarks, clients, and the AVERAGE, FREEZE, STACK, and SVD aggregation strategies.
Results
Across models, benchmarks, and aggregation methods, deeper cuts improve throughput and privacy but consistently degrade fine-tuning performance, with no method escaping the decline.
Takeaways & Limitations
Partition depth is not utility-neutral in Transformer-based SFF; near-isometric propagation and attention collapse help explain why system-optimal cuts harm quality.
Takeaways & Limitations
The analysis assumes client-side adapter aggregation is the source of any performance divergence when other conditions are identical.

Abstract

from arXiv · show

Split Federated Fine-tuning (SFF) is a promising paradigm for scaling Large Language Models (LLMs) by partitioning model depth between resource-constrained clients and a centralized server. While system incentives for throughput and privacy favor deep partitions, the impact of such configurations on model utility remains poorly understood. In this work, we identify and characterize the Depth-Performance Dilemma: the regime that maximizes system efficiency is precisely where fine-tuning quality collapses. Through a comprehensive audit across four model scales (GPT-2 to Llama-3-8B) and diverse benchmarks, we demonstrate that deeper partitions provide monotonic gains in throughput and privacy at the cost of catastrophic performance plateaus. We evaluate a suite of state-of-the-art federated adapter aggregation methods including AVG, STACK, SVD, and FREEZE, revealing that while these techniques are effective in standard Federated Learning, they fail to mitigate the artifacts unique to split architectures. Finally, we provide a mechanistic diagnosis for this failure, tracing the collapse to the near-isometric topology of Transformers, which allows aggregation noise to propagate without attenuation until it triggers Attention Collapse in the server partition. Our findings challenge the prevailing assumption that partition depth is a utility-neutral tuning knob and provide a structural foundation for stable distributed LLM fine-tuning.

1 Introduction

SFF addresses computational and privacy constraints by splitting LLM depth between clients and a shared server, but deep cuts expose a Depth-Performance Dilemma. Across the reported evaluation, throughput and privacy improve with depth while task performance degrades.

SFF lets clients run shallow layers in parallel while a Main Server processes shared deep blocks, enabling memory-feasible, privacy-aware optimization.
CNN-based intuitions about safe deep partitions do not directly transfer to Transformers, whose uniform dimensionality lacks pooling-based information attenuation.
As the cut layer deepens, throughput rises and privacy leakage decays while GPT-2 Small task performance degrades consistently.
The paper audits four model scales and three benchmarks, evaluating aggregation methods and tracing collapse to aggregation noise, near-isometric propagation, and attention collapse.

2 Related Work and Gaps

Prior split-learning work largely treated cut depth as an efficiency or privacy control, with conclusions shaped by CNN topology. The paper identifies open questions about Transformer depth sensitivity and adapter aggregation in shared-server split models.

CNN-based split-learning analyses rely on pooling-induced downsampling, whereas Transformers lack this topology-specific noise-filtering mechanism.
Recent split-LLM frameworks prioritize feasibility, latency, or privacy while tacitly treating partition depth as utility-neutral.
The comparison landscape distinguishes CNN-focused depth analyses from this paper’s systematic Transformer audit across aggregation strategies.
Federated adapter methods such as STACK, FREEZE, and SVD were developed or evaluated in standard Federated Learning rather than shared-server split settings.
The paper’s gaps are whether deeper cuts create a performance tipping point and how aggregation methods behave with sequential server updates.

3 The Depth-Performance Dilemma

SFF’s system incentives favor deeper cuts because they reduce server load and improve privacy, but the same regime consistently harms fine-tuning quality. The degradation appears across models, benchmarks, and aggregation methods and is not explained by failed convergence.

3.1 System Benefits of Deep Cuts: In the server-bound regime, throughput scales favorably as cut depth increases because each client-side Transformer block reduces sequential server load.
3.1 System Benefits of Deep Cuts: Moving from d=0.5 to d=0.9 delivers up to a 5× throughput gain, while deep cuts remain throughput-favorable or privacy-favorable under realistic settings.
3.1 System Benefits of Deep Cuts: Beyond d ≈0.8, Empirical Privacy rises sharply and reaches 100 at d > 0.9 across GPT-2 scales and Llama-3-8B-Instruct.
3.2 Impact of Depth on Fine-tuning Performance: The evaluation sweeps cut depth across four models, diverse benchmarks, and multiple client-side adapter aggregation protocols.
3.2 Impact of Depth on Fine-tuning Performance: No aggregation method avoids depth-induced degradation: FREEZE collapses most sharply, AVERAGE declines steadily, and STACK and SVD are more robust but not immune.
3.2 Impact of Depth on Fine-tuning Performance: Training loss converges in every configuration, but deeper cuts converge to worse optima, with the PPL separation emerging early and widening during training.
3.2 Impact of Depth on Fine-tuning Performance: The dilemma is that throughput and privacy improve with deeper partitions precisely where fine-tuning performance degrades and model quality collapses.

4 Diagnosing the Depth-Performance Dilemma

Deep cuts expose a transition from aggregation-invariant behavior to method-sensitive degradation. Near-isometric noise propagation and Attention Collapse explain why the shrinking server partition cannot correct incoming aggregation artifacts.

Shallow cuts: aggregation-invariant regime: At shallow cuts, aggregation choice has little impact because the long server partition can absorb federated updates.For GPT-2 Small, AVERAGE, STACK, and SVD cluster between AVG5 = 0.564 and 0.617.
Deep cuts: aggregation-sensitive regime: At deep cuts, performance diverges substantially by aggregation method across models and tasks.For GPT-2 Small, SVD achieves AVG5 = 0.570, compared with 0.528 for STACK, 0.498 for AVERAGE, and 0.310 for FREEZE.
Deep cuts: aggregation-sensitive regime: Even SVD declines from AVG5 = 0.617 at shallow depth to 0.570 at deep depth on GPT-2 Small, showing that aggregation cannot remove the residual depth penalty.The remaining gap points to a structural property of the Transformer server.
FL baseline: the limiting case: Every SFF configuration outperforms standard FL because SFF retains at least one unperturbed server layer as a buffer.At 8B scale on GSM8K, deep-cut AVERAGE, FREEZE, and SVD accuracies are 0.500, 0.517, and 0.521, versus FL baselines of 0.420, 0.420, and 0.428.
Near-isometric noise propagation: Unlike CNNs, Transformer server partitions propagate aggregation noise essentially intact because residual connections and layer normalization preserve signal norms.The perturbation ratio never drops below 1.0 across models and cut depths.
Attention Collapse: Attention Collapse removes the nonlinear capacity needed to correct noise, producing the performance cliff once attention matrices become near-identity and low-rank.Measured thresholds occur at ℓ≈6 for GPT-2 Small, ℓ≈11 for Medium, and ℓ≈19 for LLaMA, aligning with accelerated degradation.

5 Conclusion and Future Work

The study finds that deep partitions maximize throughput and privacy precisely where fine-tuning quality collapses across models, benchmarks, and aggregation methods. It attributes this collapse to aggregation noise, near-isometric propagation through Transformer residuals, attention collapse, and sequential server processing.

Conclusion: Deep partitions maximize throughput and privacy while fine-tuning quality collapses across GPT-2 and Llama-3-8B-Instruct on E2E NLG, GSM8K, and GLUE.The pattern holds under every aggregation method studied.
Conclusion: Aggregation noise severity depends on how well each method preserves the low-rank update manifold.The conclusion identifies this as the first factor in the collapse mechanism.
Conclusion: Near-isometric propagation through Transformer residual architecture delivers aggregation artifacts intact to the output.This prevents depth from acting as a utility-neutral efficiency knob.
Conclusion: Attention collapse in deep server layers eliminates corrective capacity where noise arrives most intact, compounded by sequential server processing.The conclusion presents these as additional compounding structural factors.
Experimental scope: The experiments use non-IID data, heterogeneous LoRA ranks, periodic aggregation, and exhaustive cut-layer sweeps in the reported configurations.The supplied setup specifies Dirichlet α=0.5, ranks ri ∈ {4, 8, 16}, aggregation every I=100 steps, and 4 × (L−1) runs per GPT-2 model.

A.2 GPT-2 Small on GLUE

On GLUE, the Depth-Performance Dilemma persists for GPT-2 Small with 100 clients: deeper cuts consistently reduce classification accuracy across all four aggregation methods. SVD and STACK outperform AVERAGE and FREEZE at deep cuts, while larger client participation amplifies method differences.

Dataset: MNLI is Multi-Genre Natural Language Inference and QQP is Quora Question Pairs, both cast as classification tasks.MNLI contains 393K training and 9.8K validation samples; QQP contains 364K training and 40K validation samples.
Model and training: GPT-2 Small has 12 layers and 124M parameters, with rank-16 LoRA adapters applied to query and value projections.Training uses batch size 8, learning rate 2 × 10^-4, and maximum sequence length 512.
Federated setup: The experiment uses 100 clients, samples 10 per round for 200 rounds, and partitions data non-IID with Dirichlet α=0.5.Client adapters are aggregated every I=100 local steps.
Results: Classification accuracy degrades consistently with cut-layer depth across MNLI and QQP under all four aggregation methods.The same Depth-Performance Dilemma observed in E2E experiments also holds in this 100-client GLUE setup.
Results: SVD and STACK outperform AVERAGE and FREEZE at deep cuts on both GLUE tasks.This preserves the aggregation ordering observed in the E2E primary experiments.
Results: The 100-client setup amplifies aggregation-method differences at depth relative to the 3-client setup.The passage links this pattern to STACK’s O(K) cross-task interference and AVERAGE’s O(1/K) signal dilution.

C Proof of Theorem C.1 (Throughput Scaling)

The throughput proof models synchronized SFF rounds as a bottleneck between parallel client computation and sequential server computation. In the server-bound regime, throughput rises hyperbolically with cut depth toward the crossover, but beyond it client computation dominates and throughput decreases.

Throughput scaling: In the server-bound regime d < d⋆, throughput T scales as Θ(1/(1−d)) as d approaches d⋆.The theorem assumes Transformer computational cost is linear with depth.
Throughput scaling: Moving the client partition depth ratio from d = 0.5 to d = 0.9 yields a theoretical 5× throughput increase.The increase is bounded by the server’s sequential processing capacity.
Proof setup: The client phase executes lower-model segments in parallel, while the server phase sequentially processes activations and gradients for all K clients.These phases scale oppositely in d: client time grows and server time shrinks.
Operating regime: For GPT-2 Small, Medium, and Large, the maximum valid cut ratios retaining one server layer are 11/12=0.91, 15/16=0.94, and 35/36=0.97.Whether throughput rises across the full range depends on the location of d⋆.
Operating regime: The dilemma holds when d⋆ reaches the attention-collapse onset at approximately 50–60% depth.Examples include d⋆=0.60 for ρs/ρc=2 and K=3, and d⋆=0.75 for the E2E/GSM8K setup.
Aggregation noise: Aggregation error E is defined as deviation between the actual aggregated update and the ideal multi-task manifold.The analysis distinguishes variance, interference, and bias noise across aggregation paradigms.
Aggregation noise: FREEZE creates irreducible approximation bias by constraining updates to a fixed shared subspace, leaving orthogonal task features unlearnable.The residual energy lies in the orthogonal complement of the fixed subspace.

D.1.2 Part 1: AVERAGE (FedIT [10]) (The Variance Noise)

AVERAGE aggregation introduces variance noise because averaging factorized LoRA matrices produces cross-client terms that dilute task-specific signal and interfere during inference. The resulting update becomes high-rank and flat, while signal-to-noise ratio vanishes as client count grows.

Variance Noise: The signal term scales as O(1/K) relative to each client’s update magnitude, while cross-client interference accumulates across K(K−1) terms.The section models the cross-terms as independent isotropic variables and analyzes their spectral behavior.
Variance Noise: E[B]E[A] ≠ E[BA], so averaging factorized LoRA matrices introduces algebraic cross-terms absent from the ideal aggregated update.These cross-terms form the algebraic noise analyzed in the section.
Spectrum Flattening: Averaging transforms low-rank, spiky client updates into high-rank updates with a flat spectrum and noise spread across all dimensions.The noise energy is not concentrated in a task-specific direction, unlike the original signal.
Spectrum Flattening: AVG therefore flattens sharp task-specific features into a high-rank, low-magnitude noise floor.The section names this phenomenon Spectrum Flattening.
Inference Interference: For an input from client k, the summed server adapters include interference from every other client, producing an error whose signal-to-noise ratio vanishes as K →∞.The signal shrinks faster than the accumulated interference under the stated model.

E Analysis of Spectral Denoising

Spectral denoising improves aggregated LoRA updates by retaining the signal subspace while discarding orthogonal high-rank noise. Under the stated isotropic-noise model, the theoretical SNR gain is substantial for typical LLaMA dimensions and LoRA rank.

Spectral Denoising: Spectral Truncation improves SNR by projecting orthogonal noise out of the signal subspace.The theorem models the true signal as rank r and aggregation noise as high-rank and isotropic.
Filtering Operation: Because high-rank noise spreads its energy across d dimensions, projecting onto the top-r singular vectors retains only the noise aligned with the signal.The filtering operation assumes the signal is concentrated in the top-r subspace when signal dominates noise.
Quantitative Gain: For a typical LLaMA model with d = 4096 and LoRA rank r = 16, the theoretical SNR improvement is 256×.The reported gain follows the dimension-to-rank ratio 4096/16.

F Complexity Analysis

SFF complexity depends on how Transformer blocks, LoRA adapters, and split activations are distributed across clients and the server. Local computation scales with client-side depth, while communication includes both adapter exchange and activation-gradient traffic.

Computational Complexity: Per-epoch local training cost T(·) scales linearly with the number of trainable Transformer blocks executed on the client.The client executes ℓc blocks, while the main server executes L −ℓc blocks.
Computational Complexity: Federated server costs aggregate client-side LoRA adapters, with rank terms determined by the client ranks rk and server-side rank rs.If LoRA is applied to all layers, the relevant client-depth term is replaced by L.
Communication Overhead: Each split-learning optimization step transmits cut activations and gradients of size Θ(nm) in both directions between clients and the main server.Communication is decomposed into federated LoRA exchange and bidirectional split-learning traffic.

G Empirical Privacy Evaluation via Inversion Attacks

The privacy evaluation trains adversarial token-inversion models on cut-layer representations exposed to the server, using recovery rate and normalized privacy as depth-dependent metrics. Across GPT-2 and Llama-3-8B, normalized privacy increases with cut depth and reaches a deepest-cut plateau at full privacy under the reported protocol.

Evaluation Protocol: Adversarial inversion models reconstruct input tokens from the cut-layer hidden states exposed to the server.The cut representation is the sole information the server receives about the client’s private input.
Evaluation Protocol: The evaluation covers GPT-2 Small, Medium, and Large, with 12, 24, and 36 layers respectively, under split inference.The client computes the intermediate representation at cut layer ℓc.
Inversion Model: The attacker is a token-wise MLP trained on aligned representation–token pairs with cross-entropy loss under strong white-box assumptions.The setup gives the adversary access to representation–token pairs and knowledge of the data distribution.
Metrics: Empirical Privacy is defined as EP = (1 −RR) × 100, where 0 means perfect token reconstruction and 100 means complete reconstruction failure.Recovery Rate is token reconstruction accuracy, and the metric is computed independently for each cut layer.
Llama-3-8B Extension: At 8B scale, normalized privacy rises steeply through early layers, plateaus across middle depths, and saturates at 100 at the deepest cut.Llama-3-8B-Instruct reproduces the qualitative depth–privacy trend observed for the GPT-2 family.

H Additional Experiments

Additional experiments examine convergence across cut depths and compare unified end-to-end scores across GPT-2 models, ranks, and cutlayers. Deeper cuts slow convergence and worsen final validation perplexity, while Stacking and SVD are more stable than Averaging and Freezing.

Convergence: Deeper cuts slow convergence and worsen final validation perplexity in heterogeneous GPT-2 settings.Validation perplexity is plotted against rounds for GPT-2 Small and Large across cut depths.
Convergence: Stacking and SVD are consistently more stable than Averaging and Freezing during convergence.The comparison covers GPT-2 Small and Large under heterogeneous settings.
End-to-end evaluation: Table 10 compares average E2E scores for GPT2-S, GPT2-M, and GPT2-L across cutlayers at ranks 8 and 16 under homogeneous settings.The unified E2E score averages min–max normalized ROUGE-L, METEOR, BLEU, CIDEr, and NIST scores.