Decentralized LLM Inference over Edge Networks with Energy Harvesting

• Edge deployment of increasingly capable LLMs is challenged by limited computational, memory, and energy resources.Source: §I · p. 1
• Decentralized inference distributes model workloads across devices, improving flexibility and cost-effectiveness while reducing the burden on individual nodes.Source: §I · p. 1
• Empirical measurements quantify the energy use of LLM transformer blocks on an Nvidia Jetson AGX Orin during collaborative-inference simulation.Source: §I · p. 1
• A semi-Markov model represents battery-state evolution using processing parameters and green-energy arrivals.Source: §I · p. 1
• Scheduling methods use the model to reduce device downtimes and increase distributed-inference job throughput.Source: §I · p. 1
• A dynamic power mode balances energy consumption against task completion time, supporting energy-efficient decentralized inference.Source: §I · p. 1

• The framework distributes LLM layers across device groups, replicating identical portions within groups for parallel inference.Each request selects one device from each successive group, passing intermediate outputs between groups.Source: §II · p. 2
• Devices use renewable sources and batteries, with energy arrivals modeled as node-specific uniform distributions over bounded rates.The model captures variation arising from solar-panel characteristics and local weather conditions.Source: §II · p. 2
• Each device has a one-job queue and enters power-saving downtime between lower and upper battery thresholds.This limits memory use and prevents complete battery depletion.Source: §II · p. 2
• The system optimizes device selection across heterogeneous energy consumption, processing time, and replenishment rates.It also evaluates switching among Jetson AGX Orin power modes to balance energy use and inference time.Source: §II · p. 2

System assumptions

• Each edge node combines an energy-harvesting source, a battery, discrete time slots, Bernoulli job arrivals, and statistically independent renewable-energy arrivals.Energy collected during a stage is obtained by convolving the per-slot arrival distribution according to the stage duration.Source: §III · pp. 2–3

Stage operation

• Processing stages may execute a queued job, idle without jobs, or suspend computation in power-saving mode, with duration κ_mδ determined by the selected power mode.Power-saving stages reject new jobs and continue until the battery exceeds the recovery threshold.Source: §III · pp. 2–3

State representation

• A node state is represented as S_m = (Q_m, E_m, γ_m), combining a binary queue state, discrete battery level, and active or power-saving mode in a semi-Markov chain.The state space supports analysis of the node’s statistical evolution and transitions over time.Source: §III · p. 3

Power-mode control

• Hysteretic thresholds switch nodes into power saving below E_th and restore active operation only above E′_th, while active power modes depend on available energy.This prevents unwanted oscillations in power-saving behavior and determines stage duration and job-arrival probability.Source: §III · p. 3

State transitions

• State transitions account for energy arrivals, computation consumption, queue occupancy, job arrivals, and whether the node is active or power saving.Separate transition cases cover empty queues, pending jobs, rejected arrivals during power saving, and processing stages with or without new arrivals.Source: §III · p. 3

Performance measures

• The model computes average battery energy and the probability that energy falls below a threshold, enabling threshold selection to reduce device downtime.The stationary distribution of the embedded Markov chain generally needs to be computed only once unless network parameters change.Source: §III · pp. 3–4

Scheduling policies

• Three policies—uniform, long-term, and adaptive—select job-assignment probability distributions with increasing use of energy-state information.Uniform samples available devices; long-term optimizes average energy risk; adaptive further adjusts rates using current energy availability.Source: §IV · p. 4

Static long-term optimal scheduling

• The long-term policy minimizes the risk that battery levels fall below a threshold, producing a static optimal-on-average solution over an infinite horizon.Its acceptable input rate is obtained from the stationary distribution under a known energy-arrival distribution and a user-defined tolerable risk.Source: §IV · p. 4

Algorithm description

• The system updates device energy states and power modes each slot, sets layer-specific scheduling probabilities, and then performs decentralized inference.Offline stationary-distribution computation keeps scheduling subroutines fast enough for real-time applications.Source: §IV · p. 4

Static long-term optimal scheduling

• The long-term input-rate calculation considers energy-risk minimization, whereas the algorithm operates by repeatedly updating energy, selecting power modes, assigning rates, and running inference.The supplied passage indicates that the energy-risk objective does not by itself capture every scheduling consideration.Source: §IV · p. 4

Scheduling model

• The scheduling probability selects a device within each layer or group to execute an inference run, while q_lim denotes the maximum arrival rate a device can tolerate under its energy-harvesting distribution.Because input rate is the inverse of delay, the model also incorporates processing-time constraints through the expected processing-slot quantity κ̄.Source: §IV · p. 4

Adaptive scheduling

• The adaptive policy starts from long-term input rates and decreases rates for devices in critical power mode PM1 by a quantity proportional to α/N_ℓ.The resulting rates are normalized, with probability shifted toward devices having greater energy availability.Source: §IV · pp. 4–5

Power-mode evaluation

• 47 completed jobs with the dynamic power mode, alongside an 18% average battery improvement over fixed 30 W, compared with 45 jobs and 42% battery for 30 W.Fixed 15 W completed 31 jobs with 89% battery, while fixed 60 W completed 58 jobs with 16% battery.Source: §V · p. 5
• At the 0.01 downtime-risk threshold, dynamic power tolerates an input rate of about 0.64, compared with about 0.33 for 60 W.For 15 W and 30 W, processing-time limits constrain the rates to 1/3 and 1/2, respectively.Source: §V · p. 5

Scheduling evaluation

• 15% downtime risk is achieved against uniform scheduling when varying energy arrivals, while downtime is consistently halved when varying job arrival rate.The long-term policy uses semi-Markov-derived probabilities that favor devices with higher energy income.Source: §V · pp. 5–6
• Adaptive scheduling reduces downtime probability by up to 10% versus long-term and maintains about 1% downtime even when one job arrives per time slot.The latter condition corresponds to continuously active processing under the chosen energy settings.Source: §V · p. 6

Processing-capacity evaluation

• Semi-Markov-based scheduling increases normalized throughput by about 10% at relatively low energy arrival, while adaptive scheduling adds about 2% over long-term.With varying job arrivals, long-term and adaptive drop about 3 and 7 fewer jobs than uniform scheduling, respectively.Source: §V · p. 6

VI. CONCLUDING REMARKS

• The semi-Markov model represents energy-harvesting edge computers under time and energy constraints, using parameters measured from a Jetson Orin.It is tested with decentralized LLM inference and simple scheduling algorithms.Source: §VI · p. 6
• The derived scheduling policies improve processing performance while reducing the risk of device shutdown.The method is described as adaptable to similar scenarios with different parameter settings.Source: §VI · p. 6