PELM: Power Efficient On-Device LLM Inference with Speculative Decoding and Dynamic Voltage Frequency Scaling

Weisi Yang; Stephen Xia

Source-linked AI summary

PELM: Power Efficient On-Device LLM Inference with Speculative Decoding and Dynamic Voltage Frequency Scaling

Weisi Yang, Stephen Xia

arXiv:2609.09662v1Sep 9, 2026cs.LGcs.CLcs.OScs.PF

TL;DR

On-device LLM inference is constrained by limited hardware throughput and thermal dissipation, while hardware-only DVFS leaves a difficult power–latency trade-off. PELM jointly tunes DVFS, speculative decoding, and variable verification depth, achieving up to 23.1% speedup and 52.4% lower energy consumption while maintaining comparable task performance.

Problem
On-device LLM inference faces constrained hardware, low throughput, and thermal limitations, while existing DVFS methods mainly optimize processor frequency.
Method
PELM jointly optimizes DVFS frequency, self-speculative decoding, and variable verification depth through a runtime Deep Reinforcement Learning power governor.
Results
Up to 23.1% speedup and 52.4% reduced energy consumption were achieved across platforms and datasets while maintaining task performance against existing power governors.
Takeaways & Limitations
Combining hardware frequency control with workload-specific decoding controls provides a broader mechanism for power-efficient on-device LLM inference.
Takeaways & Limitations
PELM currently depends on models exposing intermediate-layer outputs through early-exit supervision and is evaluated on NVIDIA Jetson platforms.

Abstract

from arXiv · show

Deploying Large Language Models (LLMs) directly on mobile platforms at the edge is gaining traction due to a myriad of benefits, such as increased privacy, personalization, and reduced latency. However, LLMs have heavy computational requirements, which are difficult for resource-constrained mobile and edge platforms to fulfill. In addition to limited compute resources, mobile and edge systems often have a compact form factor and lack physical mechanisms to dissipate heat generated from high processor usage rates (e.g., fans) to prevent throttling and reduced processing power, which LLMs can easily cause. To mitigate these effects, prior works have proposed various power governing strategies, such as dynamic voltage and frequency scaling (DVFS), for reducing power and heat generation for heavy computational tasks on mobile platforms. Recently, DVFS methods tailored for mobile LLMs have also been proposed. However, these methods mostly focus on optimizing hardware parameters and processor frequencies, and they fall short under some thermally constrained scenarios. Drawing from recent advances in machine learning, we identify and take advantage of the key insight that not all tokens require full-depth inference to maintain high-quality generation. Motivated by this, we present PELM, a solution that augments traditional DVFS processor frequency tuning with two additional workload-specific knobs: 1) speculative decoding and 2) variable verification depth to expand the optimization space to multiple dimensions for more power efficient on-device LLM inference. In extensive evaluations across hardware platforms and datasets, PELM demonstrates superior performance compared to state-of-the-art power governing methods, with up to 23.1% speedup and 52.4% reduction in energy consumption, while maintaining comparable task performance. The source code is available at https://github.com/imec-nu/PELM.

1 Introduction

On-device LLM inference is constrained by limited throughput and thermal headroom, while hardware-only DVFS cannot fully resolve the power–performance trade-off. PELM expands runtime control with speculative decoding and variable depth, achieving faster and more energy-efficient inference while maintaining task performance.

1 Introduction: Mobile LLM inference remains difficult because constrained hardware limits decoding throughput and compact devices dissipate heat poorly, increasing thermal-throttling risk.These constraints affect sustained high-workload inference.
1 Introduction: Hardware-only DVFS faces a dilemma: lowering frequency saves power but reduces throughput, whereas raising it worsens thermal issues.This limits further optimization through processor-frequency tuning alone.
1 Introduction: A pilot study found equivalent token speed at different layer-frequency settings, with a 24-layer model at 0.7 GHz using 41.7% less energy than a 32-layer model at 1.3 GHz.The result motivates variable-depth or early-exit execution.
1 Introduction: PELM jointly tunes DVFS frequency, self-speculative decoding, and variable verification depth through a Deep Reinforcement Learning power governor.The three knobs enlarge the optimization space and adapt runtime power and latency decisions.
1 Introduction: Up to 23.1% speedup and 52.4% reduced energy consumption were achieved across two platforms and five datasets while maintaining task performance against state-of-the-art power governors.The evaluations covered diverse workloads and thermal conditions.

2 Background, Related Works, and Motivation

Prior work addresses edge-device power or decoding efficiency separately, but PELM coordinates hardware and software controls. Its motivation is that shallower execution can preserve useful task information while reducing energy and latency.

2 Background, Related Works, and Motivation: Conventional and QoE-aware DVFS methods remain limited for LLM workloads because they primarily tune hardware behavior while facing power–latency trade-offs.Application-aware DVFS has expanded to LLMs, but existing approaches do not jointly shape decoding workload.
2 Background, Related Works, and Motivation: Self-speculative decoding reuses a truncated target model as a lightweight draft generator, avoiding the memory overhead of a separate draft model on edge devices.The draft–verify process amortizes target-model execution over multiple tokens.
2 Background, Related Works, and Motivation: A 20-layer model at 0.6 GHz matched the 10-token/s speed of a 32-layer model at 1.3 GHz while consuming 50.9% less energy.The many-to-one mapping between layer-frequency configurations reveals untapped optimization choices.
2 Background, Related Works, and Motivation: Intermediate-layer execution retained task performance within a comparable range of the final layer, although GSM8K fluctuated more across layers.This supports using shallower execution to reduce energy and latency without assuming identical quality.
2 Background, Related Works, and Motivation: PELM jointly manages DVFS frequency, self-speculation depth, verification depth, and speculative horizon under thermal and power constraints.This creates a cross-layer optimization space unavailable to hardware-only or decoding-only approaches.

3 Method

PELM formulates on-device LLM power governing as a joint optimization of DVFS, self-speculative decoding, and variable verification depth. A reinforcement-learning governor observes hardware and decoding state, selects these controls, and rewards QoE, thermal safety, low power, and efficient depth.

3.1 Problem Formulation: PELM jointly optimizes processor frequencies, self-speculation, and variable verification depth instead of tuning DVFS alone.The expanded control space targets hardware frequency, algorithmic throughput, and computational workload together.
3.1 Problem Formulation: The objective minimizes expected inference energy while meeting thermal and token-decoding-speed QoE constraints.The formulation exposes a power–latency trade-off because lower frequency reduces power while higher frequency improves generation speed.
3.1 Problem Formulation: Variable-depth self-speculation verifies tokens with vd layers rather than the full model, changing total executed layers to ns × ex + vd.The resulting speedup depends on accepted tokens, speculation count, exit layer, and verification depth.
3.2 Runtime Control: At each control step, the governor monitors utilization, temperature, power, token rate, and acceptance statistics before selecting DVFS and decoding actions.The closed loop executes early-exit drafting and variable-depth verification, while HQCache supports efficient layer skipping.
3.2 RL Formulation: A DQN converts the multi-objective control problem into reinforcement learning, using rewards that approximate the scalarized energy, QoE, thermal, utilization, and depth objectives.The Q-function is updated with the Bellman equation, and the state/action design captures processor conditions and decoding parameters.
3.2 Reward and Control: The reward encourages target-speed decoding, low GPU frequency, high speculative speedup, thermally appropriate depth, and reduced power consumption.Its exploration policy increases frequencies with thermal headroom, reduces overspeed, and adjusts speculation based on observed speedup.

4 Evaluation Setup

PELM is evaluated on diverse mobile-scale tasks and LLM sizes against built-in, speculative-decoding, and DVFS-based baselines. The evaluation measures energy, decoding speed, task quality, and performance per joule.

Datasets and LLMs: The study samples 40 examples from each of five datasets spanning math, question answering, code generation, translation, and summarization.The evaluated models are LLaMA-3.2-1B, LLaMA-3.1-8B, and LLaMA-2-13B.
Evaluation scope: The evaluation uses diverse hardware and thermal conditions to test PELM across distinct computational workloads and thermal stress levels.The selected models run on Jetson platforms, including AGX Orin and Orin Nano.
Baseline methods: PELM is compared with built-in DVFS, standard self-speculative decoding, FUSE, and zTT baselines.These baselines represent default governors, static speculation, search-based mobile-LLM DVFS, and reinforcement-learning DVFS, respectively.
Evaluation metrics: Energy per query and token decoding speed are the primary overall-performance metrics.Task-specific quality metrics include Accuracy, EM, pass@1, BLEU, and ROUGE-L; TSA summarizes normalized task quality, while PPJ divides TSA by energy.

5 Results

Across workloads and thermal conditions, PELM jointly adapts frequency, speculative decoding, and model depth to reduce energy while preserving speed, robustness, and task quality. Its benefits are strongest for moderate-to-heavy workloads, while locality effects are more complex for the 1B model on Orin Nano.

5.1 Power Governing Performance: 29.0% to 52.4% energy savings on LLaMA-13B were achieved versus four baselines, while PELM maintained the highest average generation speed under three of four cooling conditions.For 8B models, PELM consumed 13.4% to 32.5% less energy per query with comparable generation speed.
5.1 Power Governing Performance: 100% completion rates were achieved by PELM and zTT in all but one thermal scenario, whereas non-thermal-aware methods fell as low as 19% completion on the 13B workload.The single failure occurred for the extreme 13B workload under minimum cooling.
5.2 Task Performance: Task scores remained statistically comparable to vanilla full-model decoding for 8B and 1B models, with the largest degradation below 17.2%.The 13B model retained meaningful quality relative to the full-depth vanilla baseline.
5.2 Task Performance: 45.4% higher performance-per-joule than baselines was achieved in moderate-to-heavy workloads with comparable output quality at target speed.PPJ measures normalized task quality divided by energy consumption.
5.3 Ablations: Removing variable depth reduced PPJ and speed, while removing speculative decoding made every tested scenario slower and more energy intensive.At 8B Moderate, autoregressive ablation A2 consumed 31680 J versus 28059 J for PELM; removing frequency control failed on moderate-to-heavy workloads.
5.4 Query-Locality Impact: 18 of 24 larger-locality cases increased speculation speedup relative to randomized order, but 1B Orin Nano showed non-monotonic speedups and greater task-score variance.The authors attribute this complexity to the smaller model making correct speculation more challenging, while observing no catastrophic degradation.
Overhead Analysis: PELM’s CPU-only policy selection took 0.087–0.922 ms at 10 Hz, while HQCache loading was 46–60× faster than recomputing a decoder layer.The policy network has 5,687 parameters and approximately 22 KB in FP32.

6 Discussion

PELM’s applicability is bounded by model, hardware, quantization, and deployment conditions. The authors identify broader platform support, quantized evaluation, concurrent workloads, and input-aware control as future directions.

Model support: PELM requires early-exit supervision, with robust publicly available support currently concentrated in the LLaMA family.Without early-exit capability, PELM disables verification-depth control but retains DVFS, speculation, and QoE-aware regulation.
Quantization: Quantized models were not evaluated because variable-depth execution and HQCache resumption require additional preparation and validation.The authors describe quantization as complementary and leave rigorous quantized evaluation for future work.
Platform scope: Evaluation is confined to NVIDIA Jetson platforms, so other SoCs may require remapped control knobs and retuned policy targets.The authors specifically cite heterogeneous CPU/GPU/NPU governors as a portability consideration.
Deployment conditions: PELM has not been evaluated with concurrent background workloads, where system governors and services may introduce contention and frequency-scaling interactions.The authors propose joint control or coordination mechanisms for multi-tenant edge environments.
Future directions: Broader deployment may require integrating orthogonal techniques such as quantization and memory offloading across smartphones, Raspberry Pi-class boards, and NPUs.Such extensions would also require adaptation to heterogeneous DVFS controls and thermal envelopes.

7 Conclusion

PELM co-designs DVFS and speculative decoding for power-efficient on-device LLM inference. Variable-depth verification enlarges the optimization space, and evaluations across datasets, devices, hardware constraints, and QoE requirements verify its efficiency.

7 Conclusion: PELM co-designs DVFS and speculative decoding for power-efficient on-device LLM inference, adding variable-depth verification for further efficiency gains.The framework bridges decoding-agnostic DVFS methods and hardware-agnostic speculative decoding.