PaperlayerView exact revision on arXiv

Source-linked AI summary

Just Talk Once: Communication-Efficient Split Federated LLM Fine-Tuning on Edge Devices

Jiaxiang Geng, Xianhao Chen, Bing Luo

arXiv:2609.01457v1cs.DC

TL;DR

Edge LLM fine-tuning is difficult under device, bandwidth, connectivity, and data-locality constraints, while U-shaped SFT adds bidirectional communication and continuous client participation. The paper introduces L-shaped SFT and its one-shot mode, achieving lower communication and latency with competitive quality, though finite caches make the activation loss approximate.

  • Problem

    Edge-generated LLM fine-tuning must preserve target locality while operating under limited device resources, bandwidth, connectivity, and continuous-participation constraints.

  • Method

    L-shaped SFT uses weight tying to supervise server-side hidden activations with target embeddings, while one-shot SFT uploads activations once for continued server optimization over cached representations.

  • Results

    L-shaped SFT remains highly comparable to U-shaped SFT while reducing MMLU communication by 25% to 66% and latency by 34% to 70%.

  • Takeaways & Limitations

    The designs make split LLM fine-tuning more practical for bandwidth-constrained and intermittently connected edge environments.

  • Takeaways & Limitations

    With a finite server history cache, A-Loss approximates standard cross-entropy and is no longer exactly equivalent to it.

Abstract

from arXiv · show

Large language model (LLM) fine-tuning is increasingly shifting toward data generated on edge devices, where memory, computation, bandwidth, and connectivity constraints make conventional federated learning difficult to sustain. Split federated fine-tuning (SFT) improves client-side efficiency by offloading most model parameters and computation to the server but requires step-by-step bidirectional communication loop across the split interface and forces continuous client involvement throughout training. In this paper, we present L-shaped SFT, a split fine-tuning framework that removes this bidirectional bottleneck. Our key insight is that weight tying in modern LLMs enables server-side hidden activations to be directly supervised using target embeddings, allowing the training loss to be computed on the server without returning server outputs to the client. To further eliminate the need for continuous client participation, based on L-shaped SFT, we introduce one-shot SFT, in which clients upload activations once and then go offline while the server continues optimization over cached representations. We implement our design in a real system testbed with heterogeneous edge clients, including commercial smartphones and NVIDIA developer boards. Experiments demonstrate that our schemes significantly reduce communication costs and client online time compared with existing SFT baselines.

1 Introduction

Edge-generated data makes private, resource-aware LLM fine-tuning important, but existing federated and split designs impose substantial device and communication burdens. L-shaped SFT moves supervision server-side, while one-shot SFT removes continuous client participation.

  • Motivation: Edge-native fine-tuning addresses projected public-dataset exhaustion and avoids collecting sensitive device data centrally.The motivation includes ownership, deployment, and compliance constraints in domains such as finance and healthcare.
  • Existing limitations: Conventional federated learning requires clients to train full LLMs locally and transmit full-model updates, creating memory, computation, and communication bottlenecks.These requirements are particularly challenging for resource-constrained edge devices.
  • Existing limitations: SFT offloads most parameters and computation to the server but standard and U-shaped variants still require target handling and ongoing client involvement.U-shaped SFT preserves target locality by returning server outputs to clients for loss computation, creating a bidirectional training loop.
  • Proposed framework: L-shaped SFT uses weight tying to supervise server-side hidden activations with target embeddings, eliminating the activation-return and gradient-upload loop.Loss computation therefore moves to the server while target tokens remain local.
  • Proposed framework: One-shot SFT lets each client upload activations once, go offline, and leave the server optimizing cached representations.This reduces communication frequency to a single upload window and minimizes client online time in unstable edge environments.
  • Evaluation: The paper implements the designs on heterogeneous edge hardware and reports communication savings with competitive fine-tuning quality.The testbed includes commercial smartphones, NVIDIA developer boards, and a GPU server, with experiments on WikiText-2 and MMLU.

2 Background and Related Work

Prior split-learning systems reduce client resource demands but either expose targets to the server or require a step-by-step bidirectional loop. L-shaped SFT is positioned as a unidirectional alternative for intermittently connected edge devices.

  • Background: Split learning reduces client computation and memory by offloading deeper LLM layers to a server.Related systems include SplitLoRA and SflLLM, which reduce trainable state on resource-constrained devices.
  • SFT variants: Standard SFT keeps a lightweight client prefix but assumes server-side supervision, which can expose sensitive autoregressive target tokens.U-shaped SFT preserves label locality by moving the prediction head and loss computation back to the client.
  • Communication bottleneck: U-shaped SFT requires four cross-interface transmissions per training step: activation upload, activation download, gradient upload, and gradient download.The added activation-download and gradient-upload paths create its mandatory bidirectional loop.
  • Communication bottleneck: Because loss computation remains client-side, U-shaped SFT requires clients to wait for server responses and stay online throughout training.This tight synchronization is undesirable when bandwidth, connectivity, and device availability fluctuate.
  • Positioning: L-shaped SFT redesigns the split pipeline into a unidirectional protocol that reduces per-step communication while keeping target tokens local.The framework also naturally enables a one-shot mode that minimizes client online time.

3 System Design

L-shaped SFT moves supervision to the server using target embeddings and weight tying, removing U-shaped SFT’s activation-return/gradient-upload loop. This unidirectional design supports activation-space training and the one-shot mode, where clients upload once before server optimization continues.

  • Weight tying and supervision: Weight tying links the output head to the input embedding matrix, allowing hidden states to be compared directly with target embeddings.The paper writes W_head = E^⊤ and reports high embedding self-recovery across representative LLM families.
  • L-shaped SFT overview: L-shaped SFT redesigns split training so the server supervises hidden activations using target embeddings while target tokens remain local to clients.The client uploads cut-layer activations and target embeddings; the server computes the loss without receiving raw target token IDs.
  • Activation-space objective: The activation-based objective uses a contrastive loss between each server-side hidden state and its target embedding, with cached embeddings providing negative samples.The history cache Q stores target embeddings from previous steps or other clients; similarity may use a dot product or cosine similarity.
  • Activation-space objective: Under weight tying, τ = 1, and a full-vocabulary cache, the proposed loss recovers the standard cross-entropy gradient at the hidden-state interface.The cache-normalized probability becomes the standard Softmax probability under these conditions.
  • One-shot SFT: One-shot SFT lets each client perform a single local forward pass, upload activation–target-embedding pairs once, and leave while the server optimizes cached representations.This operating mode targets unstable edge environments by decoupling data provision from server-side optimization.

4 Theoretical Analysis

The analysis establishes when A-Loss preserves standard cross-entropy's optimization signal and characterizes the gradient error introduced by a finite history cache. These results justify server-side activation supervision while identifying cache coverage as the key approximation factor.

  • Gradient Consistency of A-Loss: A-Loss exactly matches standard cross-entropy's hidden-state gradient when temperature is 1, the cache covers the full vocabulary, and weight tying holds.Under these conditions, cache-normalized probabilities equal Softmax probabilities and target embeddings equal output-head vectors.
  • Gradient Consistency of A-Loss: The same gradient equality extends to the split interface, so L-shaped SFT can replace U-shaped training without changing the ideal-case training signal.The cut-layer gradient is obtained by backpropagating the server-side gradient through the server submodel.
  • Approximation Error under Finite Cache: Finite caches turn A-Loss into a cache-restricted approximation whose gradient deviation is explicitly characterized rather than exactly equivalent to full-vocabulary cross-entropy.The exact equivalence no longer holds when the server cache contains only a subset of vocabulary items.
  • Approximation Error under Finite Cache: The approximation error has two sources: renormalization over cached tokens and missing probability mass from uncached vocabulary items.The first arises from differing normalization domains; the second reflects omitted tokens with non-negligible full-Softmax probability.
  • Approximation Error under Finite Cache: A-Loss remains close to standard cross-entropy when the cache contains dominant competing tokens, while full-vocabulary coverage removes both error terms.Cache quality determines how closely the split-interface gradient approximates the full-vocabulary signal.

5 Implementation

The system uses heterogeneous edge devices and software stacks to evaluate split federated LLM fine-tuning across text generation and question-answering tasks. Experiments cover full fine-tuning and LoRA fine-tuning with models selected according to device memory and performance requirements.

  • Real-world testbed: The real-world testbed combines two NVIDIA Jetson Orin Nano boards, three Huawei nova 9 Pro smartphones, and a Wi-Fi 6 router.The server uses a Dell PowerEdge T640 with two NVIDIA A800 GPUs.
  • Software and measurement: Communication uses Flower orchestration, the Flower C++ SDK, Wireshark traffic measurement, and device-specific PyTorch, Transformers, or MobileFineTuner software.Smartphones run MobileFineTuner, while Jetson devices use PyTorch and Transformers.
  • Embedding privacy: Target embeddings can be protected by a client-private reversible transformation matrix before upload.The server cannot directly recover embeddings or tokens without the locally retained transformation matrix.
  • Data tasks: The evaluation uses WikiText-2 for text generation and partitions it into five client subsets, producing a non-IID federated distribution.Perplexity measures next-word prediction ability on this task.

6 Results

The results evaluate L-shaped SFT against the widely adopted U-shaped SFT pipeline as the primary baseline. The comparison retains the shared original U-shaped pipeline while removing auxiliary designs from prior systems for framework-level fairness.

  • Baseline: U-shaped SFT is the primary baseline for evaluating the proposed L-shaped SFT framework.The baseline represents the shared pipeline used in EUSFL, MPSL, FlexP-SFL, and MobiLLM.
  • Evaluation design: The comparison removes method-specific auxiliary designs from prior systems and retains only their shared original U-shaped pipeline.This isolates framework-level differences between U-shaped and L-shaped SFT.
  • Evaluation design: The experiments assess the proposed framework against U-shaped SFT under a common split-training evaluation setup.The stated comparison focuses on the original U-shaped pipeline rather than auxiliary system features.

6.1 Overall Comparison with U-shaped SFT

L-shaped SFT preserves fine-tuning quality comparable to U-shaped SFT while reducing communication and latency. One-shot SFT further minimizes system cost and client participation by uploading embeddings once and continuing optimization on the server.

  • Full fine-tuning: 25%–66% communication reduction and 34%–70% latency reduction are achieved by L-shaped SFT on MMLU under full fine-tuning.Performance remains highly comparable to U-shaped SFT across MMLU and WikiText-2.
  • Full fine-tuning: 25% communication reduction and 34% latency reduction are achieved consistently by L-shaped SFT on WikiText-2 under full fine-tuning.Both L-shaped and U-shaped SFT outperform the corresponding pretrained models.
  • LoRA fine-tuning: 87% communication reduction and 89% latency reduction are reached by L-shaped SFT on WikiText-2 in LoRA settings.Across 1B–8B models, final quality remains competitive with U-shaped SFT; some Llama settings exceed 70% savings.
  • One-shot SFT: One-shot SFT achieves the lowest communication cost, lowest overall latency, and shortest client active time among the compared system modes.Its communication accounting includes client-side model parameter upload and download.
  • Repeated local traversals: Increasing local data traversals improves communication and latency efficiency for both L-shaped SFT and one-shot SFT, with one-shot SFT maintaining the largest reduction.L-shaped SFT reuses cached target embeddings, while one-shot SFT completes subsequent optimization entirely on the server.

6.2 Ablation Studies

Ablations examine client-side layer placement, temperature, and dropout robustness. Communication remains stable across shallow-layer configurations, while deeper client allocation can exceed device memory; one-shot SFT is most resilient to dropout.

  • Different number of client-side layers: Activation-and-gradient communication remains unchanged when client layers increase from Emb + Head to Emb + Head + 2 Blks.Final accuracy fluctuates only slightly under these configurations.
  • Different number of client-side layers: Emb + Head + 6 Blks causes an out-of-memory failure on the client device.The result constrains excessively deep client-side allocation on memory-limited edge devices.
  • Impact of τ: τ = 1 is theoretically principled for A-Loss, and empirical performance is best when τ is set around 1.At τ = 1, the loss recovers the standard cross-entropy gradient form under the stated ideal conditions.
  • Impact of dropout: One-shot SFT is the most robust to client dropout, while U-shaped SFT degrades more sharply and L-shaped SFT degrades more gracefully as dropout increases.One-shot SFT is largely unaffected after a client successfully uploads all required embeddings once.

7 Conclusion

L-shaped SFT moves supervision to the server to remove U-shaped SFT’s bidirectional bottleneck, while one-shot SFT compresses client participation into a single upload. Testbed results indicate improved practicality for bandwidth-constrained and intermittently connected environments, with activation privacy left unaddressed.

  • L-shaped SFT moves supervision to the server, removing conventional U-shaped SFT’s bidirectional bottleneck while maintaining competitive fine-tuning quality.The design substantially reduces communication costs and overall latency.
  • One-shot SFT compresses client participation into a single upload, after which optimization proceeds entirely on the server.This minimizes client active time.
  • Real-world tests on heterogeneous edge devices show that both designs make split LLM fine-tuning more practical in bandwidth-constrained and intermittently connected environments.
  • The paper does not address potential privacy leakage from uploaded activations, which remains a common challenge for split fine-tuning and split federated learning frameworks.The authors identify activation-level differential privacy and homomorphic encryption as orthogonal protections.
Loading 2609.01457v1…