Source-linked AI summary
Resource Allocation for Stable LLM Training in Mobile Edge Computing
Chang Liu, Jun Zhao
TL;DR
Mobile devices cannot efficiently train LLMs despite demand for local, privacy-preserving AI, motivating collaborative mobile-edge training. The paper splits PEFT across mobile users and edge servers, jointly optimizing energy, delay, and stability; simulations report reduced energy and delay with improved LLM stability.
Problem
LLM training demands computational resources beyond standard mobile devices, while mobile deployment seeks local processing for real-time, personalized, and privacy-preserving applications.
Source: §1 · p. 1Method
The paper assigns PEFT of initial LLM layers to mobile users, trains later layers at edge servers, and optimizes energy, delay, and stability using fractional programming and related procedures.
Source: §1 · p. 2Results
Simulations indicate that the proposed approach reduces energy usage and delay while enhancing LLM stability across mobile edge computing settings.
Source: §5.3 · p. 8, §6 · p. 9Takeaways & Limitations
Collaborative mobile-edge training provides a resource-aware framework for improving the efficiency and stability of LLM training under mobile computing constraints.
Source: §1 · p. 2, §6 · p. 9The analysis assumes the loss function is L-Lipschitz and strongly convex, and neglects edge-server downlink transmission energy because it is considered negligible.
Source: §3.4 · p. 4, §3.2 · p. 4
Abstract
from arXiv · showhide
As mobile devices increasingly become focal points for advanced applications, edge computing presents a viable solution to their inherent computational limitations, particularly in deploying large language models (LLMs). However, despite the advancements in edge computing, significant challenges remain in efficient training and deploying LLMs due to the computational demands and data privacy concerns associated with these models. This paper explores a collaborative training framework that integrates mobile users with edge servers to optimize resource allocation, thereby enhancing both performance and efficiency. Our approach leverages parameter-efficient fine-tuning (PEFT) methods, allowing mobile users to adjust the initial layers of the LLM while edge servers handle the more demanding latter layers. Specifically, we formulate a multi-objective optimization problem to minimize the total energy consumption and delay during training. We also address the common issue of instability in model performance by incorporating stability enhancements into our objective function. Through novel fractional programming technique, we achieve a stationary point for the formulated problem. Simulations demonstrate that our method reduces the energy consumption as well as the latency, and increases the reliability of LLMs across various mobile settings.
1 INTRODUCTION
The paper motivates collaborative LLM training in mobile edge computing because mobile devices offer privacy benefits but lack the computational resources required by LLMs. It proposes dividing training between mobile users and edge servers while jointly optimizing energy, delay, and model stability.
1 INTRODUCTION
- LLMs support advanced mobile applications, but their substantial computational requirements exceed the capabilities of standard mobile devices.Local processing can improve privacy and data security, yet mobile hardware remains resource-constrained.Source: §1 · p. 1
- Mobile edge computing brings computation closer to data sources, reducing reliance on centralized infrastructure while supporting local processing and improved response times.Source: §1 · p. 1
- The proposed framework assigns initial-layer fine-tuning to mobile users and computationally intensive later-layer training to edge servers.This division uses PEFT methods to make local training more suitable for mobile environments.Source: §1 · pp. 1–2
- Because distributed fine-tuning can create model instability, the paper incorporates stability into the objective to reduce performance variation across training instances.The motivation is to preserve reliable behavior under local-data and distributed-training fluctuations.Source: §1 · p. 2
- The paper derives an upper bound for average-replace-one stability and decomposes resource optimization into fractional programming and CCCP subproblems.The fractional programming step targets offloading and resource allocation, while CCCP addresses user-to-edge association.Source: §1 · p. 2
2 RELATED WORK
Related work covers MEC resource allocation, PEFT versus in-context learning, and stability in fine-tuned LLMs. The paper distinguishes its approach by jointly considering energy and delay, using average-replace-one stability, and combining mobile-edge LLM training with optimization methods offering stationary-point guarantees.
Resource allocation in mobile edge computing
- Prior MEC studies optimize offloading, latency, energy, or resource allocation using exhaustive search, relaxation, alternating optimization, or reinforcement learning.The reviewed methods include approaches with high complexity, decoupled variables, or no local or global optimality guarantee.Source: §2 · p. 2
- Unlike work focused only on energy, this paper includes delay in the objective, making resource allocation more complex but jointly addressing both criteria.Source: §2 · p. 2
PEFT vs. In-Context Learning (ICL)
The model stability of fine-tuned large language models
- Stability research includes Mixout regularization and theoretical analyses of sparse fine-tuning as a regulatory mechanism for pretrained models.Source: §2 · p. 2, Figure 1 · p. 3
- The paper uses average-replace-one stability to assess overall performance variation after replacing one training instance, rather than pointwise prediction sensitivity.This measure is presented as better aligned with network-wide reliability and efficiency in edge computing.Source: §2 · p. 3
3 SYSTEM MODEL
The system model splits LLM training between mobile users and edge servers, while jointly accounting for computation, communication, energy, delay, and model stability. It then converts stability into an upper-bound-constrained optimization problem over layer partitioning and resource-allocation variables.
3 SYSTEM MODEL
- Mobile users fine-tune the first α_n transformer layers, then send intermediate results to an associated edge server for processing of the remaining Υ−α_n layers.The model considers N users and M edge servers training LLMs with the same architecture.Source: §3 · p. 3, §3.1 · p. 3, §3.2 · p. 4
3.1 Local Computation Model
- Local computation models user-layer training through GPU frequency, core count, FLOPs per cycle, cubic power consumption, and corresponding computation energy.The local model also accounts for transmission of intermediate results using user power, allocated bandwidth, and channel gain.Source: §3.1 · p. 3
3.2 Edge Computation Model
- Edge computation accounts for the frequency, core count, and per-core capability of each server GPU, while combining remaining-layer delay and energy in a weighted cost.Downlink transmission energy is omitted because edge-server power capacity is substantially higher and its transmission energy is considered negligible relative to LLM training energy.Source: §3.2 · p. 4
3.3 LLM Stability
- Average-replace-one stability measures how an individual prediction changes when a training sample is replaced, supporting reliability across users’ variable local data.The paper uses this measure to represent LLM stability in the system objective.Source: §3.3 · p. 4
3.4 Problem Formulation
- The joint problem optimizes local-layer counts, user-server associations, transmission power, bandwidth, and user and edge GPU frequencies while weighting delay, energy, and stability.Because the resulting MINLP is non-convex, fractional programming solves resource variables through parametric convex problems, while CCCP handles associations iteratively.Source: §3.4 · p. 4
- Theorem 1 bounds the expectation of the loss’s average-replace-one stability by 2L^2/[k(1−α)] when fine-tuning proportion α of parameters, making model stability measurable and enabling reformulation into P2.The result assumes an L-Lipschitz, strongly convex loss function and replaces the sum of users’ stability terms with their upper bounds.Source: §3.4 · p. 4
4 PROPOSED ALGORITHM
The proposed algorithm decomposes the non-convex resource-allocation problem into alternating optimization and CCCP-based subproblems. These transformations enable iterative solution procedures that converge to a stationary point for the relevant formulations.
4.1 Optimizing 𝜶, 𝒑, 𝒃,𝒇𝑼,𝒇𝑬given 𝝌
- The 𝜶 variables are relaxed from discrete to continuous values during optimization and rounded to the nearest integer afterward.This relaxation is used because the discrete variable 𝜶_n is difficult to handle directly.Source: §4.1 · p. 5
- Fractional programming transforms P3 into a sequence of P4 problems, which alternate optimization solves using convex subproblems.The method alternates optimization of 𝜶, 𝒑, 𝒃, 𝒇𝑼, 𝒇𝑬 and the auxiliary variables 𝒛, 𝝂, 𝒒 from an initial feasible point.Source: §4.1 · p. 5
- The alternating-optimization process for P4 is non-increasing and yields a stationary point for P3 after convergence.P4 is solved through successive updates of the resource variables and auxiliary variables, with convergence determined by an objective-function tolerance.Source: §4.1 · pp. 5–7
4.2 Optimizing 𝝌given 𝜶, 𝒑, 𝒃,𝒇𝑼,𝒇𝑬
- For fixed 𝜶, 𝒑, 𝒃, 𝒇𝑼, and 𝒇𝑬, the second subproblem replaces discrete variables with continuous ones and addresses the remaining non-convex constraint through penalization.The resulting P6 is an indefinite quadratic, difference-of-convex problem handled by CCCP and first-order Taylor approximation.Source: §4.2 · p. 7
- Repeated CCCP runs from different feasible starting points generate local optima, after which the solution with the smallest average value is selected.This multi-start procedure is used because directly solving P6 for a feasible P5 solution may not always be viable.Source: §4.2 · p. 7
5 SIMULATIONS
Simulations evaluate the collaborative training method under default settings, weighting-factor changes, and varying user and edge-server populations. The proposed method reduces energy and delay, maintains strong stability, and converges to a stationary point across tested configurations.
5 SIMULATIONS
- The simulations use a MEC network with 50 mobile users, 10 edge servers, and Meta’s LLaMA-7B model with 32 transformer layers by default.The setup also specifies mobile and edge GPU capabilities, bandwidth, transmission power, batch size, hidden-state dimensionality, and input-token lengths.Source: §5 · p. 7
5.1 The performance of the proposed collaborative training method
- The proposed collaborative method reduces energy consumption and delay relative to edge-server and local training approaches.Its delay is closer to edge-server training while significantly outperforming local training.Source: §5.1 · pp. 7–8, Figure 2 · p. 8
5.2 The performance of proposed algorithms under weighting factors
- The proposed approach consistently achieves the lowest energy consumption and delay, and the highest model stability, across weighting-factor settings.The comparison varies each weighting factor from 1 to 10 while holding the other two at 1.Source: §5.2 · p. 8, Figure 3 · p. 9
5.3 The impact of the number of users and the number of edge servers
- Across different user quantities, the proposed association method outperforms greedy and random strategies in both energy consumption and average delay.Greedy association can overload computationally inefficient servers, while random association performs worst overall.Source: §5.3 · p. 8, Figure 5 · p. 9
- With 100 users, the algorithm reaches a stationary point and converges within 13 iterations for all tested numbers of edge servers.Fewer edge servers produce faster convergence because the optimization has fewer constraints and parameters.Source: §5.3 · pp. 8–9, Figure 4 · p. 9
6 CONCLUSION
The paper presents collaborative LLM training in which mobile users train initial layers and edge servers train later layers. Its multi-objective method targets energy, delay, and stability, using stability analysis, fractional programming, and CCCP; simulations report improvements on all three objectives.
6 CONCLUSION
- The collaborative framework assigns initial-layer training to mobile users and computationally intensive later-layer training to edge servers.This partitions training between mobile and edge resources.Source: §6 · p. 9
- The optimization reduces overall energy usage and user latency while improving LLM stability.The paper also establishes an upper bound for average-replace-one stability.Source: §6 · p. 9
- The proposed algorithm combines fractional programming and the Concave-Convex Procedure to derive solutions.The conclusion reports these methods as the solution approach for the formulated optimization problem.Source: §6 · p. 9
A PROOF OF THEOREM 1
The proof analyzes fine-tuning stability by relating the regularized objective to parameter changes between neighboring datasets. Under strong convexity and Lipschitz assumptions, it derives a bound on the change in the learned parameters and uses it to establish the stability result.
A PROOF OF THEOREM 1
- The fine-tuning objective includes a regularization term that penalizes deviations from pretrained parameters, providing a lower bound for the original problem.The term is ∥(I−M)(w−w0)∥2, where w0 denotes pretrained-model parameters.Source: A PROOF OF THEOREM 1 · p. 9
- The proof defines fS(w)=LS(w)+(1−α)∥w−w0∥2 and uses strong convexity to relate objective differences to parameter-distance differences.The argument evaluates the objective at minimizers obtained from neighboring datasets.Source: A PROOF OF THEOREM 1 · pp. 9–10
- The Lipschitz assumption on the loss converts the parameter-distance bound into a stability bound for individual losses.The proof combines the Lipschitz relation with the strong-convexity inequality.Source: A PROOF OF THEOREM 1 · p. 10
- The resulting parameter sensitivity is bounded by 2L/((1−α)k) for neighboring datasets.The bound is stated for every i in {1, ..., k}.Source: A PROOF OF THEOREM 1 · p. 10