Empirical Guidelines for Deploying LLMs onto Resource-constrained Edge Devices

• The study addresses how resource constraints influence personalized edge-LLM design and the trade-offs affecting learning efficiency and accuracy.Source: §1 · pp. 1–2
• It systematically evaluates combinations of model types and sizes, customization methods, historical data, compression, learning time, and downstream-task difficulty.Source: §1 · pp. 1–2
• Task difficulty determines whether parameter learning or RAG is preferable: easy and hard tasks favor parameter learning, while moderate tasks favor RAG.Source: §1 · p. 2
• More historical user data is not always better; fine-tuning often should stop early once the model reaches a stable position.Source: §1 · p. 2
• Distillation offers the most stable compression performance, quantization the highest peak performance with less stability, and pruning is unsuitable for edge LLMs.Source: §1 · p. 2
• The work presents an exploratory study covering the full range of constraints involved in deploying LLMs on edge devices.Source: §1 · p. 2

• The evaluation specifies edge-device constraints, resource measurements, personalization datasets, selected LLMs, compression methods, customization methods, and default experimental settings.Source: §2 · p. 2

3.1 OPTIMAL MODEL AND CUSTOMIZATION FOR LLMS ON THE EDGE

• Across four tasks, LoRA consistently outperforms the other three PEFT methods, occasionally underperforms RAG, and consequently motivates subsequent investigations focused on LoRA and RAG.Overly small models may be incapable of handling even simple classification tasks and should be avoided.Source: §3.1 · pp. 3–4
• Task difficulty determines the preferred edge customization: LoRA suits simple and challenging tasks, whereas RAG with a strong model suits intermediate difficulty.For simple classification, small models with LoRA are preferred; as difficulty rises, the choice shifts toward RAG, then back to LoRA for challenging summarization.Source: §3.1 · pp. 3–4

3.2 CHOICE OF LORA STRATEGIES

• Fixing alpha is generally more useful than varying it, and settings such as (16, 16) or (16, 32) work in most edge cases.Increasing rank or alpha does not necessarily improve performance, even for larger models deployed with 10G or 16G RAM.Source: §3.2 · p. 5

3.3 TREND OF LORA TRAINING TIME FOR EDGE LLMS

• More LoRA training data does not necessarily improve results: 3–4 hours is usually sufficient on the A14 Bionic chip, with longer training consistently helping only the easiest task.The experiments vary training from 1 to 8 hours; consistent improvement appears only on LaMP-2, the easiest of seven tasks.Source: §3.3 · p. 6

3.4 IMPACT OF USER HISTORY DATA VOLUME ON RAG PERFORMANCE

• Increasing RAG history from 100 to 1000 samples usually provides no significant gain, and can reduce performance for some Pythia models.Across eight models, RAG performance is more related to the LLM’s internal reasoning capabilities than to the amount of stored history.Source: §3.4 · p. 7

3.5 COMPARISON OF MODEL COMPRESSION TECHNIQUES ON EDGE LLMS

• Distillation is the most stable compression approach, quantization can reach the highest peak performance, and pruning is generally unsuitable for edge LLMs.Phi models offer robust performance, GPTQ can work especially well for summarization, and shearing preserves performance less efficiently in RAM than quantization.Source: §3.5 · pp. 8–9

3.6 COMPARISON BETWEEN COMPRESSED AND UNCOMPRESSED EDGE LLMS

• With limited personalized data, quantized models can outperform unquantized models on difficult tasks, whereas pruning causes broad performance drops.Model size is also task-dependent: quantized 2B–3B models are often sufficient for classification, while larger models help summarization requiring more semantic understanding.Source: §3.6 · p. 10

4 CONCLUSION AND FUTURE DIRECTIONS

• Edge deployment must balance model size, customization strategy, training time, and resource constraints rather than maximize any single factor.The study emphasizes that larger trainable parameter counts and longer training are not always optimal, while compressed models can sometimes adapt faster.Source: §4 · p. 10

A.1 EDGE DEVICE CONSTRAINTS

• RAM constrains edge learning because fine-tuning requires model weights, optimizer state, and gradients, whereas RAG additionally requires storing user-data embeddings.This makes the memory trade-off differ between parameter learning and retrieval-based customization.Source: §A.1 · p. 17

A.2 RUNNING LLMS ON EDGE DEVICES

• The evaluation models edge-device training by profiling data samples processed per hour and using those rates to represent different training durations.This simulation supports experiments across models, devices, and LoRA settings without directly training on every edge device.Source: §A.2 · pp. 17–18

A.3.1 BACKGROUND

• The benchmark uses personalized LaMP histories and queries spanning classification and text-generation tasks, with seven datasets covering citation, tagging, rating, headline, title, email, and tweet tasks.These user-specific datasets are intended to test whether an edge LLM can learn from historical data and answer personalized prompts.Source: §A.3.1 · p. 19

A.3.2 DATA PREPARATION

• The evaluation averages performance over 100 randomly selected users and profiles models against edge-device RAM capacities and training peak memory.Selected models span multiple families and are chosen to fit 4G, 6G, 8G, 10G, and 16G RAM settings.Source: §A.3.2 · p. 19, §A.4 · p. 20
• Task difficulty is normalized across datasets using random-guessing accuracy for classification, while summarization uses raw accuracy.Higher normalized accuracy indicates an easier task; the reported values range from 0.995 to 5.157 for classification and 0.1460 to 0.3475 for summarization.Source: §A.3.2 · pp. 19–20

A.5 MODEL COMPRESSION

• The compression study covers quantization, pruning, and distillation, using GPTQ for quantization and Sheared-LLaMA for pruning, while Phi represents distillation.Quantization reduces weight precision and model size; distillation trains a smaller model to emulate a larger one.Source: §A.5 · pp. 20–21

A.6 METHODS FOR EDGE LLM CUSTOMIZATION

• PEFT updates a small portion of model parameters, whereas RAG retrieves relevant user-history information without fine-tuning but stores embeddings in memory.The experiments focus on LoRA for PEFT and vary LoRA rank and alpha, while RAG varies user-history size and retrieval settings.Source: §A.6 · p. 21

A.7 EXPERIMENTAL SETTINGS

• The experiments evaluate 100 users per dataset with up to 1000 history documents, using standardized generation settings and MIPS retrieval for RAG.RAG embeds history documents with all-MiniLM-L6-v2 and retrieves the top three results.Source: §A.7 · pp. 21–22

B.1.1 EXPERIMENTS: EXAMINE RAG AND PEFT

• The experiments compare PEFT and RAG across a wide range of edge LLMs and resource settings.The comparison is reported in the study’s performance tables alongside zero-shot baselines.Source: §B.1.1 · p. 25, Table 7 · p. 25, Table 8 · p. 26
• The evaluated PEFT configuration uses rank = 8, alpha = 16, and 8 training hours, with performance compared directly against RAG.The table covers performance comparisons for selected LLMs across the customization methods.Source: Table 7 · p. 25
• The experiments also provide zero-shot performance comparisons for all selected LLMs as a reference condition.The zero-shot results are reported separately from the PEFT and RAG comparison.Source: Table 8 · p. 26

B.1.2 EXPERIMENTS: TRAINABLE PARAMETERS BASED ON FIXED ALPHA AND VARYING RANK

• Figures 10–13 map performance across LaMP-1 to LaMP-7 as trainable parameters and training hours vary for four Pythia model sizes.The models are Pythia-410m, Pythia-1b, Pythia-1.4b, and Pythia-2.8b.Source: §B.1.2 · p. 27, Figure 10 · p. 27, Figure 11 · p. 27
• The heatmaps fix alpha = 32 and vary LoRA rank across 8, 16, 24, 32, 40, 48, 56, 72, and 80.The y-axis represents trainable parameters associated with rank values, while the x-axis represents training hours.Source: §B.1.2 · p. 27, Figure 10 · p. 27, Figure 11 · p. 27

B.1.3 EXPERIMENTS: TRAINABLE PARAMETERS BASED ON VARYING ALIGNED ALPHA AND RANK

• The aligned alpha-and-rank experiments correspond to Section 3.2.Source: §B.1.3 · p. 29
• Performance heatmaps vary trainable-parameter count and training hours across LaMP-1 to LaMP-7 for Pythia-410m, Pythia-1b, Pythia-1.4b, and Pythia-2.8b.The experiments use alpha = rank values of 8, 16, 24, 32, 40, 48, 56, 72, and 80.Source: Figure 14 · p. 29, Figure 15 · p. 29, Figure 16 · p. 30

B.1.4 EXPERIMENTS: COMBINATIONS OF SEVERAL COMMONLY USED ALPHA AND RANK

• The experiments with common alpha-and-rank combinations correspond to Sections 3.2 and 3.3.Source: §B.1.4 · p. 31
• Performance is compared across seven datasets for StableLM-2-1.6b and StableLM-3b-4e1t using different common rank and alpha combinations.Both experiments use the default settings.Source: Figure 18 · p. 31, Figure 19 · p. 32

B.1.5 EXPERIMENTS: MODEL SIZE AND TRAINING TIME

• The model-size and training-time experiments correspond to Section 3.3.Source: §B.1.5 · p. 33
• Performance is compared across difficulty-varied datasets for Pythia models from 410m to 2.8b parameters paired with 4G to 16G RAM.The selected configurations are Pythia-410m on 4G, Pythia-1b on 6G, Pythia-1.4b on 8G, StableLM-1.6b on 10G, and Pythia-2.8b on 16G RAM.Source: Figure 20 · p. 33
• A second comparison evaluates OPT-125m through OPT-2.7b on 4G, 6G, 8G, and 16G RAM across datasets with different difficulty levels.The figure represents training durations from 1 to 8 hours.Source: Figure 21 · p. 34

B.2.1 EXPERIMENTS: RAG PERFORMANCE ON DIFFERENT SIZES OF USER HISTORY DATA

• RAG performance improvement is evaluated for models of different sizes across varying user-history data sizes on all seven datasets.The experiments include four Pythia models and four additional models: Llama-3b-GPTQ, StableLM-2-1.6B, TinyLlama-1.1B-GPTQ, and TinyLlama-1.1B.Source: Figure 22 · p. 35, §B.2.1 · p. 35, Figure 23 · p. 36

C.1 EXPERIMENTS: QUANTIZATION MODELS

• Quantization experiments examine Gemma-2b and Gemma-7b using Tables 9 and 10, then compare quantized Llama models and same-size models with different architectures.These experiments correspond to Section 3.5 and provide supplemental material on LLM quantization.Source: §C.1 · p. 37, Table 9 · p. 37, Table 10 · p. 37
• Figure 24 compares quantized Llama models of different sizes across seven datasets as training-data size increases.The experiments use rank = 8 and alpha = 32 for LoRA.Source: Figure 24 · p. 38
• Figure 25 compares different quantized models with the same size across seven datasets as training-data size increases.The experiments use rank = 8 and alpha = 32 for LoRA.Source: Figure 25 · p. 39

C.2 EVALUATION LLMS PRUNING

• Pruning experiments compare pruned and unpruned Sheared-LLaMA models across data sizes, LoRA settings, model sizes, and training durations.The comparisons include TinyLlama, Llama-3b, and Llama-3b-GPTQ across seven datasets.Source: §C.2 · p. 40, Table 11 · p. 40, Table 12 · p. 40

C.3 EVALUATION LLMS DISTILLATION

• Knowledge-distillation experiments evaluate Phi-1.5, Phi-2, and Phi-3 under different LoRA settings, data sizes, and training durations.Results are organized in comparisons under eight-hour training and across one to eight hours.Source: §C.3 · p. 44, Table 15 · p. 44, Table 16 · p. 45

C.4 EXPERIMENTS: COMPARE THE THREE COMPRESSION TECHNIQUES

• Compression-technique experiments compare a wide range of compressed models using a common heatmap scale to identify optimal models and conditions for each task.The evaluated models include GPTQ, pruned, distilled, and other compressed variants across seven datasets and eight alpha-rank combinations.Source: §C.4 · p. 46, Figure 28 · p. 46, Figure 29 · p. 46

C.5 EXPERIMENTS: COMPARE THE COMPRESSED MODEL WITH UNCOMPRESSED MODELS

• Compressed-versus-uncompressed experiments compare pruned and unpruned Sheared-LLaMA models and quantized versus unquantized Gemma-2b across one to eight training hours.The section includes comparisons under fixed and varied LoRA settings.Source: §C.5 · p. 51, Table 17 · p. 51, Table 18 · p. 52

D.1 LARGE LANGUAGE MODELS CUSTOMIZATION FOR DOWNSTREAM TASKS

• Fine-tuning updates LLM behavior using targeted datasets, commonly through LoRA, while prompt engineering includes in-context learning and retrieval-augmented generation without changing model weights.RAG retrieves relevant documents and adds their knowledge to prompts.Source: §D.1 · p. 56

D.2 LARGE LANGUAGE MODEL EVALUATIONS

• Existing benchmarks such as MMLU, GPQA, HumanEval, and GSM-8K are described as unsuitable for edge LLMs, so the paper uses the LaMP dataset.LaMP is selected because it closely aligns with common edge-LLM use cases.Source: §D.2 · p. 56
• Prior optimization studies examine quantization, fine-tuning choices, transformer architectures, and LLM selection, but do not fully provide a guidebook for edge LLM constraints.The stated gap concerns evaluation work focused on cloud LLMs or incomplete coverage of the constraints introduced for edge deployment.Source: §D.2 · p. 56