PaperlayerView exact revision on arXiv

Source-linked AI summary

PLM: Efficient Peripheral Language Models Hardware-Co-Designed for Ubiquitous Computing

Cheng Deng, Luoyang Sun, Jiwen Jiang, Yongcheng Zeng, Xinjian Wu, Wenxin Zhao, Qingfa Xiao, Jiachuan Wang, Haoyang Li, Lei Chen, Lionel M. Ni, Haifeng Zhang, Jun Wang

arXiv:2503.12167v2cs.CL

TL;DR

Edge devices have limited resources, and smaller language models can still strain their storage and bandwidth. PLM co-designs model architecture and edge systems, combining MLA with squared ReLU and a multi-phase training approach. The paper reports that PLM surpasses existing small language models trained on public data and runs on devices including mobile phones and Raspberry Pis.

  • Problem

    Edge devices have limited resources, and smaller language models can retain architectural demands that strain device storage and bandwidth.

    Source: §1 · pp. 1–2

  • Method

    PLM co-designs an edge-oriented model and system, using MLA and squared ReLU alongside multi-phase training and fine-tuning.

    Source: §1 · pp. 2–3, §2 · p. 4, §4.3 · p. 9

  • Results

    PLM surpasses existing small language models on standard benchmarks and runs on devices ranging from PCs to mobile phones and Raspberry Pis.

    Source: §9 · p. 21

  • Takeaways & Limitations

    The paper frames peripheral language models as a co-design field integrating hardware and algorithms, and releases PLM-1.8B model weights for further research.

    Source: §1 · p. 3, §9 · p. 21

    MLA reduces memory use but adds computation for latent-vector decompression, so its suitability depends on the target hardware’s memory and computational balance.

    Source: §8.2 · p. 20

Abstract

from arXiv · show

While scaling laws have been continuously validated in large language models (LLMs) with increasing model parameters, the inherent tension between the inference demands of LLMs and the limited resources of edge devices poses a critical challenge to the development of edge intelligence. Recently, numerous small language models have emerged, aiming to distill the capabilities of LLMs into smaller footprints. However, these models often retain the fundamental architectural principles of their larger counterparts, still imposing considerable strain on the storage and bandwidth capacities of edge devices. In this paper, we introduce the PLM, a Peripheral Language Model, developed through a co-design process that jointly optimizes model architecture and edge system constraints. The PLM utilizes a Multi-head Latent Attention mechanism and employs the squared ReLU activation function to encourage sparsity, thereby reducing peak memory footprint during inference. During training, we collect and reorganize open-source datasets, implement a multi-phase training strategy, and empirically investigate the Warmup-Stable-Decay-Constant (WSDC) learning rate scheduler. Additionally, we incorporate Reinforcement Learning from Human Feedback (RLHF) by adopting the ARIES preference learning approach. Following a two-phase SFT process, this method yields performance gains of 2% in general tasks, 9% in the GSM8K task, and 11% in coding tasks. In addition to its novel architecture, evaluation results demonstrate that PLM outperforms existing small language models trained on publicly available data while maintaining the lowest number of activated parameters. Furthermore, deployment across various edge devices, including consumer-grade GPUs, mobile phones, and Raspberry Pis, validates PLM's suitability for peripheral applications. The PLM series models are publicly available at https://github.com/plm-team/PLM.

1 Introduction

PLM addresses the challenge of running language models on edge devices through hardware–model co-design, combining an efficient architecture with multi-phase training and peripheral deployment. The paper presents PLM-1.8B as competitive with similarly sized open-source-trained models and describes experiments across edge hardware.

1 Introduction

  • Edge devices share limited memory and computing resources with other applications, while KV-cache offloading can add I/O overhead.The paper notes that models may need to be loaded only when required and unloaded to release memory.Source: §1 · p. 2
  • PLM applies modeling and system co-design, with squared ReLU for computational sparsity and Multi-Head Latent Attention for attention efficiency.The paper describes its model design as combining squared ReLU with low-rank attention.Source: §1 · p. 2
  • The report covers data curation, training, evaluation, and deployment, including multi-phase training with WSDC, progressive supervised fine-tuning, and preference learning.It also identifies fast inference on peripheral devices as a focus.Source: §1 · p. 3
  • PLM-1.8B combines multi-head latent attention and squared ReLU, with competitive performance against similarly sized LLMs trained on open-source data and efficient decoding.The paper presents these features as part of a model tailored for peripheral devices.Source: §1 · pp. 2–3
  • The authors report implementing PLM on consumer-level GPUs, Jetson Orin, mobile phones, laptops, and Raspberry Pis.They frame peripheral language-model development as a co-design problem spanning hardware and language modeling.Source: §1 · p. 3

2 Architecture

PLM uses a decoder-only Transformer with MLA to reduce KV-cache memory and squared ReLU activations in a streamlined, ungated FFN. Its design retains full-rank queries and balances memory use against inference latency.

2 Architecture

  • PLM uses a decoder-only Transformer with MLA, Decoupled RoPE, a Qwen2-inherited BPE tokenizer, squared ReLU activations, and tied input-output embeddings.The model has 0.31B embedding parameters and 1.51B non-embedding parameters.Source: §2 · p. 4
  • MLA compresses key and value projections into a shared low-rank latent space, then reconstructs the vectors during computation to reduce KV-cache memory.PLM omits low-rank compression on query projections.Source: §2 · pp. 4–5
  • PLM preserves full-rank query projections because query compression adds inference computation, which the authors consider undesirable for latency-sensitive applications.The model also integrates Decoupled RoPE.Source: §2 · p. 5
  • The streamlined FFN omits gating, while squared ReLU supports sparse computation by activating positive inputs and setting nonpositive inputs to zero.The paper says ReLU2 can allow inactive neurons to be skipped and can reduce feedforward I/O overhead through similar activation patterns across consecutive tokens.Source: §2 · p. 5
  • Combining low-rank attention with sparse activations balances KV-cache reduction and computational load, trading memory cost against latency.The authors report architecture-search experiments focused on attention-head count and hidden dimension when finalizing PLM.Source: §2 · p. 5

3 Data

PLM’s data strategy combines curated open-source and synthetic pre-training data across three phases with progressive supervised fine-tuning and preference training. The section describes dataset selection, filtering, and the language scope of post-training corpora.

3 Data

  • PLM’s pre-training corpus combines open-source data with chain-of-thought, RAG-augmented, and large-model-distilled data across three phases.The phases are described as general-knowledge learning, integrating high-quality data, and learning from internal knowledge-intensive datasets.Source: §3 · p. 5

3.1 Data Collection

  • Open-source datasets are selected for quality, volume, and domain coverage; scoring models support sampling, and MinHash removes duplicates from the Chinese corpus.The scoring model uses normalized DCLM and CCI3-HQ scores to score datasets without intrinsic quality metrics; the authors note processing-time and efficiency constraints leave room for improvement.Source: §3.1 · p. 5
  • An internal pre-training dataset is generated using synthetic data, drawing on Cosmopedia and extended chain-of-thought reasoning [34] [35].The passage frames synthetic data as a response to data shortages and a way to steer models toward specialized or robust capabilities.Source: §3.1 · p. 5

3.2 Data for Pre-Training

  • Phase 1 pre-trains on 1.65 trillion tokens across English, Chinese, code, and mathematics datasets.The listed sources include DCLM, Chinese CCI3-HQ and MAP-CC, StarCoder, Proof-Pile, and MathPile.Source: §3.2 · p. 6
  • Phase 2 uses 550B tokens, adding higher-quality general data and instruction data, including extended CoT, QA, and generated knowledge content.The passage gives a 5:2:2 ratio for Chinese general, English general, and additional content, and lists further code, math, and academic sources.Source: §3.2 · p. 6
  • Phase 3 uses a smaller, more diverse corpus, including over 60M internal instruction samples, reasoning data, and math data, while selectively down-sampling general datasets.The passage identifies FineWeb-Edu and Chinese-FineWeb-Edu-V2 among the down-sampled sources.Source: §3.2 · p. 6

3.3 Data for Post-Training

  • Post-training SFT progresses from 407M samples across 17 general-domain datasets to 131M samples across 30 datasets spanning safety, math, coding, and knowledge-intensive instruction.Some Phase 1 datasets are retained in Phase 2 for distributional balance.Source: §3.3 · p. 7
  • Preference training combines offline general-preference data with on-policy UltraFeedback prompts and CodeUltraFeedback for a second training round.PLM’s instruction and alignment corpora are exclusively in Simplified Chinese and English.Source: §3.3 · p. 7

4 Pre-Training Details

PLM’s training details cover architecture search, multi-phase pre-training, progressive SFT, and ARIES-style preference training. The section reports phase-one validation loss and describes changes to data quality and learning-rate schedules across training.

4.1 Model Architecture Search

  • Architecture search evaluates FLOPs, MACs, and empirical performance on a specially selected 100B-token sample from the final pre-training corpus.Table 5 is identified as reporting architecture-search experiments.Source: §4.1 · pp. 7–8, Table 5 · p. 8

4.2 Pre-training

  • PLM is trained on 2.48 trillion tokens with WSDC: warm-up and stable learning, decay, then a constant-rate phase over the final 280 million tokens.The constant phase uses a 4,096-token context and up-samples high-quality data while adding instruction-like data.Source: §4.2 · p. 8
  • The authors also describe a phase-three cosine-decay schedule at 3e-5 over an additional 280 billion tokens to refine the model.They associate the extension with mathematics, coding, and encyclopedic knowledge.Source: §4.2 · p. 8
  • Phase-one validation loss exceeded 2.3, prompting the authors to use higher-quality data in later phases.The passage contrasts this with a cited threshold below 2.2 for the emergence of most abilities [40].Source: §4.2 · p. 8, Figure 6 · p. 9

4.3 Supervised Fine-tuning

  • SFT uses two progressive full-parameter phases, moving from general-purpose instruction data toward specialized math, coding, and other domains to produce PLM-Instruct.Figure 7 presents the training curves for the two-phase SFT.Source: §4.3 · p. 9, Figure 7 · p. 9

4.4 Preference Training

  • The authors adopt ARIES self-refinement loss for preference training to strengthen self-refinement, addressing reported degradation after repeated self-refinement in open-source LLMs.The loss is also described as having an auxiliary effect on direct answering ability in experiments.Source: §4.4 · pp. 9–10
  • Preference training first uses offline data, then combines 5K UltraFeedback prompts with CodeUltraFeedback for a second round using a reward model to construct preferences.The offline dataset is llama-3.1-tulu-3-70b-preference-mixture, and the reward model is RLHFlow/ArmoRM-Llama3-8B-v0.1.Source: §4.4 · p. 10

5 Evaluation: Training Results

PLM is evaluated through phased training, benchmark comparisons with similarly sized models, and assessments of its task performance. The results report competitive overall performance, with particular strengths in coding, reasoning, and mathematical tasks.

5.1 Evaluation and Experiment Setup

  • The evaluation covers language modeling, knowledge, mathematics, coding, and commonsense benchmarks, using specified zero-shot, 5-, 10-, and 25-shot settings.Coding was evaluated with EvalPlus; the other listed tasks used the LM-Evaluation Harness.Source: §5.1 · pp. 10–11

5.2 Evaluating Model Performance during Training

  • Perplexity decreased across DCLM, MAP-cc, StarCoder, and ProofPile through three-phase pre-training, reaching 6.75 on ProofPile in Phase 3.The passage notes substantial gains for StarCoder and ProofPile.Source: Table 6 · p. 11, §5.2 · p. 11
  • Phase 2 SFT averaged 57.00, with GSM8K at 60.80 and HumanEval at 57.90.GSM8K improved by 26 points over earlier phases; MBPP temporarily dipped in Phase 1 before recovering.Source: Table 7 · p. 11, §5.2 · p. 11
  • Online RLHF and PLM-Instruct averaged 55.99 and 56.94, respectively; GSM8K reached 63.50 and HumanEval 64.63.Identity RL underperformed relative to Online/Offline RLHF, while MathQA showed minimal gains.Source: §5.2 · p. 11, Table 8 · p. 12

5.3 Evaluating PLM and the baselines

  • PLM-1.8B trailed Qwen2.5-1.5B on MMLU, CMMLU, and C-Eval, while its 1.8B parameters and 32 layers compare with Yulan-Mini's 56 layers.The authors state that PLM would offer significantly lower inference latency; it performed slightly worse than Yulan-Mini in coding and logical reasoning.Source: §5.3 · p. 13
  • PLM-1.8B scored 57.29 on average, third behind Qwen2.5-1.5B (59.25) and Yulan-Mini-2.4B (57.51).It led HumanEval and ranked second in ARC-C, ARC-E, MBPP, and BoolQ.Source: §5.3 · p. 12, Table 10 · p. 13

6 Evaluation: In the Context of Modeling-Hardware Co-Design

The evaluation compares model efficiency across architectures, hardware platforms, and quantization settings. PLM is reported to perform strongly across FP16 and quantized configurations, with a cited speed advantage over Fox in two Snapdragon 8 Gen 3 settings.

6 Evaluation: In the Context of Modeling-Hardware Co-Design

  • The efficiency analysis uses architecture comparisons to examine MLA's and sparsity's speed impact, alongside layer-count and model-size comparisons.The selected comparisons include MiniCPM-4B versus GLM-Edge-4B for MLA, and Nemotron-4B versus GLM-Edge-4B for sparsity.Source: §6 · p. 13
  • The deployment evaluation spans an NVIDIA A10, Jetson, Intel CPU, Apple M3, Snapdragon 8 Gen 3, and Broadcom BCM2712 hardware.The BCM2712 is the SoC used in Raspberry Pi 5.Source: §6 · p. 14
  • Q8 and Q4 models generally have lower memory cost than FP16 counterparts, but latency reductions vary by model architecture and hardware.The evaluation describes Snapdragon 8 Gen 3 and M3 as competitive in specific settings.Source: §6 · p. 14
  • PLM is reported to outperform Fox on Snapdragon 8 Gen 3 in FP16 generation and 8-bit quantization prefill, while showing strong optimization across FP16, 8-bit, and 4-bit settings.The comparison concerns models with the same layer count and PLM's deployment across diverse hardware platforms.Source: §6 · p. 14, Table 12 · p. 15, Figure 8 · p. 16

7 Analysis: Insights into the Design of PLM

The analysis examines how MLA and sparse activations affect PLM’s efficiency on edge hardware. It reports high activation sparsity and advantages in selected edge inference settings, while noting hardware-dependent trade-offs.

7.1 Learned the Attention in PLM

  • MLA reduces KV-cache memory use, but its extra prefill computation can be mitigated by hardware acceleration on modern edge processors.The analysis also links decode efficiency to reduced memory I/O transfer time.Source: §7.1 · pp. 16–17
  • MLA reduces KV-cache use but increases prefill computation; its decoding-complexity difference from GQA depends on sequence length.When I/O latency dominates, the authors report that MLA outperforms GQA.Source: §7.1 · p. 16
  • On Snapdragon, PLM shows advantages over same-depth GQA model Fox in the reported settings, including longer text sequences.The passage also reports that MLA retains advantages in edge settings where tensor parallelism is less important.Source: §7.1 · p. 17, Figure 9 · p. 17

7.2 Learned the Sparsity in PLM

  • PLM’s sparse-activation design is presented as broadly effective for edge deployment, where zero computations can be optimized.The analysis discusses offloading when models cannot fit fully in GPU memory or system RAM.Source: §7.2 · pp. 17–18
  • With llama.cpp loading one layer at a time, experiments report clear PLM design benefits in resource-constrained inference.The experiment concerns offloading and iterative layer loading.Source: §7.2 · p. 18, Figure 10 · p. 18
  • The sparsity study plots the perplexity difference between sparse and dense models against sparsity rate, treating full sparsity without quality loss as unattainable.The figure is described as showing how architectures differ in sensitivity to increased sparsity.Source: Figure 11 · p. 18, §7.2 · p. 19
  • PLM’s sparsity rate reaches 90.9% in MLP and 74.3% overall, indicating fewer actively computed parameters.The authors connect zero-valued activations to potential hardware acceleration.Source: §7.2 · p. 19

8 Discussion: Future Works and Direction

The discussion frames peripheral language-model development as a co-design problem spanning algorithms and systems. It outlines future adaptation, learning, collaboration, and capability directions, while emphasizing trade-offs and device requirements.

8 Discussion: Future Works and Direction

  • The authors identify integration of modeling and system development as crucial to designing and training peripheral LLMs.Their discussion draws on PLM design, training, and deployment cases.Source: §8 · p. 19, §8.1 · p. 19

8.1 Future Works

  • Planned on-device adaptation includes sparse updates, quantization-aware low-precision fine-tuning, and memory-efficient gradient accumulation across sessions.The stated aim is personalized language models on resource-constrained edge devices.Source: §8.1 · p. 19
  • Future work proposes privacy-preserving federated learning and edge-cloud task allocation, with simple real-time queries handled at the edge and complex tasks delegated to the cloud.The proposed federated framework includes differential privacy, distillation, and secure aggregation.Source: §8.1 · pp. 19–20
  • The authors propose expanding PLM toward multimodal capabilities and improving SLM reasoning through distillation, fine-tuning, reinforcement learning, and inference-time scaling.The discussion also names hallucination mitigation and multi-agent scenarios as future directions.Source: §8.1 · p. 20

8.2 Direction

  • MLA’s memory savings come with added decompression computation, so its suitability depends on the target hardware’s balance of memory and processing capacity.The discussion says this trade-off should be evaluated for each edge deployment scenario.Source: §8.2 · p. 20
  • The proposed direction for sparse LLMs emphasizes algorithm–hardware co-design, including hardware-aware sparsity and system-level optimization.The discussion names sparse data formats, memory architectures, and compiler optimizations among the areas for future development.Source: §8.2 · pp. 20–21
  • Peripheral deployment priorities include rapid response, privacy, resource efficiency, and power efficiency under device constraints.The authors describe these as requirements distinct from cloud-based implementations.Source: §8.2 · p. 21

9 Conclusion

The conclusion presents PLM as a co-designed edge language model and reports benchmark performance and deployment across several device types.

9 Conclusion

  • PLM combines Multi-head Latent Attention and squared ReLU, outperforms existing small language models on standard benchmarks, and runs on PCs, mobile phones, and Raspberry Pis.The authors release PLM-1.8B-Base and PLM-1.8B-Instruct and frame peripheral LLM development as hardware–algorithm co-design.Source: §9 · p. 21

A Architecture Searching Experiments

The appendix describes architecture-search experiments and summarizes training data, configurations, and loss curves across pre-training and supervised fine-tuning phases.

A Architecture Searching Experiments

B.1 Pre-Train Dataset

  • Table 19 presents the pre-training data distribution and allocation.The passage identifies the table’s scope but provides no distribution values.Source: Table 19 · p. 28

B.2 Internal Pre-Train Dataset

  • Collected supervised fine-tuning datasets are restructured into pre-training data.The collection is listed in Table 22.Source: §B.2 · p. 28, Table 22 · p. 29

B.3 Post-Train Data

C Training Details

  • Table 23 summarizes pre-training hyperparameters across three phases, including learning-rate schedules, weight decay, batch sizes, and optimization parameters.The stated purpose is to support convergence across training stages.Source: C Training Details · p. 30, Table 23 · p. 30

C.1 Pre-training

  • Figure 12 depicts validation loss against training tokens across the pre-training phases.The text says phase-specific trends reflect different pre-training strategies.Source: §C.1 · p. 30, Figure 12 · p. 30

C.2 Supervised Fine-tuning

  • Table 24 details configurations for two SFT phases, while Figure 13 depicts loss reduction and convergence differences across phases.The configurations include batch sizes, learning-rate schedules, and regularization techniques.Source: §C.2 · pp. 30–31, Table 24 · p. 31, Figure 13 · p. 31

C.3 Preference Training

This section describes PLM’s two-phase preference training through its training setup and the evolution of reward accuracy and training loss.

C.3 Preference Training

  • The RLHF hyperparameter table covers batch sizes, learning rates, weight decay, Adam parameters, and coefficients for policy updates and reward shaping.The named RLHF-related coefficients are α, βDPO, and βentropy.Source: §C.3 · p. 31, Table 25 · p. 31
  • Phase 1 shows greater fluctuations in the implicit reward accuracy curve, while phase 2 trends upward more smoothly.The paper describes this as progressive refinement of the reward signal during alignment with human preferences.Source: §C.3 · p. 32, Figure 14 · p. 32
  • Training loss fluctuates in phase 1 and gradually stabilizes in phase 2 during preference-based learning.Table 25 is identified as displaying the RLHF training curves.Source: §C.3 · p. 32, Figure 15 · p. 32

D Computational Analysis

The section analyzes computational costs for Multi-head Latent Attention (MLA) and grouped-query attention (GQA) across attention definitions, prefill, and decoding. It describes sequence-length scaling and the roles of cache memory and hardware throughput.

D Computational Analysis

  • MLA stores compressed latent key-value states for inference, reducing KV-cache requirements relative to storing the full KV matrix, while adding computational overhead.The analysis frames this as a memory-computation trade-off and examines MLA and GQA computational costs.Source: D Computational Analysis · p. 33

D.1.1 Definition

  • MLA forms latent key-value states through low-rank compression, combines content and RoPE-based components, and projects the concatenated head outputs into the final output.The passage defines the compressed state, per-head key, value, and query components, and the final output projection.Source: §D.1.1 · p. 33

D.1.2 Prefill

  • MLA prefill combines linear-in-sequence-length projection and RoPE costs with quadratic attention interactions and an output projection.The analysis identifies the dependence on sequence length N and model dimensions in the total cost.Source: §D.1.2 · p. 34

D.1.3 Decode

  • MLA decoding caches joint compression states, while attention over the sequence contributes a cost linear in N; latency depends on hardware I/O and computational throughput.The total decode cost combines projections, attention, and output projection.Source: §D.1.3 · p. 34

D.2.1 Definition

  • GQA defines grouped key and value representations alongside per-head queries, then combines head outputs through a linear output projection.Its stated definition uses key-value heads and a shared total head dimension.Source: §D.2.1 · pp. 34–35

D.2.2 Prefill

  • GQA prefill includes projection and RoPE costs, quadratic attention interactions, and a final output projection.The analysis describes the quadratic attention term as dominant alongside linear contributions from projections and RoPE.Source: §D.2.2 · p. 35

D.2.3 Decode

  • GQA decoding caches historical keys and values, with attention cost scaling linearly in N; latency depends on hardware throughput and trades memory footprint against parallelizable computation.The stated decode cost also includes key-value and query projections, RoPE, and an output projection.Source: §D.2.3 · p. 35

E Activation Sparsity Determination Experiments

The section presents an experiment for selecting MLP activation sparsity rates against a target perplexity reduction. It also shows PLM-generated examples in math, code, emotional communication, and logic in Chinese and English.

E Activation Sparsity Determination Experiments

  • The algorithm searches candidate activation sparsity rates and returns one only if the modified model meets the target perplexity reduction of ∆ppl = 1.It computes baseline perplexity, masks activations using a threshold, and evaluates the modified model on a benchmark dataset; otherwise, it reports no suitable rate.Source: E Activation Sparsity Determination… · p. 36

F Chat Generations from PLM

Related papers

Loading 2503.12167v2…