PaperlayerView exact revision on arXiv

Source-linked AI summary

HybridInfer: Thermal-Aware Reinforcement-Learning Tier Routing for On-Device, Edge, and Cloud LLM Inference

Simran Koul

arXiv:2609.30270v1cs.LG

TL;DR

Existing multi-tier LLM routers do not use the thermal state of the phone executing local inference, while sustained on-device generation can destabilize the mobile GPU runtime. HybridInfer routes among on-device, edge, and cloud models using thermal headroom and query complexity; on Android, it achieved significantly higher quality than hand-tuned heuristics at the lowest cost among adaptive conditions.

  • Problem

    Multi-tier routers generally ignore the physical thermal state of the phone running the local model, despite sustained on-device inference being thermally constrained and prone to runtime instability.

    Source: §I · p. 1

  • Method

    HybridInfer uses an offline-trained Q-learning policy to select among on-device, edge, and cloud tiers from phone thermal headroom and query complexity, with a reward that includes a locality bonus.

    Source: §I · p. 1, C. Router state, actions, and baselines · p. 3, F. Reinforcement-learning router · p. 4

  • Results

    The learned router achieved significantly higher answer quality than two hand-tuned heuristics at the lowest cost of any adaptive condition.

    Source: §I · p. 2

  • Takeaways & Limitations

    Explicitly valuing locality is a precondition for thermal-aware routing to remain meaningful, and routing addresses on-device reliability and coverage as well as performance.

    Source: §VI · p. 7

    The study uses a single device model, and quality gaps among tiers are modest on the tested workload.

    Source: D. Limitations · p. 7

Abstract

from arXiv · show

On-device inference with small language models keeps user data local, works offline, and incurs no per-query cost, so the on-device tier is preferred when it is adequate. It is thermally constrained, however, and I find the constraint is sharper than a slowdown: on a flagship Snapdragon device, sustained on-device generation destabilizes the GPU inference runtime, which crashes or silently wedges after a few consecutive queries. The failure lies in the current toolchain (OpenCL kernel compilation and long-prompt prefill on the mobile GPU), recurs even when the device is cool, and is worst for long generations. Multi-tier routers across on-device, edge, and cloud models can relieve this pressure, but existing routers are thermal-blind and typically evaluated in simulation or on non-mobile hardware. I present HybridInfer, a thermal-aware reinforcement-learning router for a three-tier hierarchy (on-device Llama 3.2 3B, edge Llama 3.1 8B with retrieval, cloud GPT-4o) that uses the phone's thermal headroom and a query-complexity estimate as state and selects a tier by an offline-trained Q-learning policy. Its reward trades quality against latency, cost, and a thermal penalty, plus a locality bonus crediting on-device execution. I show this bonus is a precondition for thermal-aware routing: without it the optimal policy offloads every query. On a real Android benchmark of 210 prompts, the learned router attains significantly higher quality than two hand-tuned heuristics (paired Wilcoxon, p < 0.02) at the lowest cost of any adaptive condition. Always-on-device conditions match per-query quality on servable queries but are three to six times slower and fail on long queries, so routing wins on latency, reliability, and coverage rather than quality. To my knowledge this is the first use of on-device thermal headroom to select among LLM inference tiers of differing capability on real hardware.

I. INTRODUCTION

HybridInfer addresses the mismatch between mobile inference’s local advantages and its thermal and reliability constraints by routing across on-device, edge, and cloud tiers. The introduction argues that thermal-aware routing must value local execution and reports improved quality over hand-tuned heuristics at the lowest cost among adaptive conditions.

I. INTRODUCTION

  • Sustained on-device generation can crash or wedge the GPU inference runtime after a few queries, including when the phone is cool.The reported causes lie in the MLC-LLM and Adreno OpenCL stack, including kernel-compilation and long-prompt prefill failures.Source: §I · pp. 1–2
  • Existing multi-tier routers use query difficulty or answer confidence but do not account for the executing phone’s physical state, and close systems were evaluated off-phone.The paper contrasts this gap with measurement studies that treat thermal throttling as a performance ceiling [13], [14].Source: §I · p. 1
  • The router combines phone thermal headroom and query complexity to select among on-device, edge, and cloud tiers, with a locality bonus that keeps local execution preferable when thermally feasible.Without the bonus, faster edge and cloud tiers dominate and the policy offloads every query, making thermal state irrelevant.Source: §I · pp. 1–2
  • HybridInfer’s contributions include thermal-headroom tier selection on real Android hardware and a reproducible workload of 210 prompts with frozen gold references.The paper also frames sustained on-device instability as a reliability and coverage concern, not only an efficiency concern.Source: §I · p. 2
  • HybridInfer achieved significantly higher answer quality than two hand-tuned routing heuristics at the lowest cost of any adaptive condition.The reported results come from three replications on an unplugged, uncooled device.Source: §I · p. 2

II. RELATED WORK

Prior LLM routers allocate queries by difficulty, confidence, or model strength, while multi-tier device–edge–cloud systems establish the architecture. The paper distinguishes HybridInfer by using a real smartphone’s thermal state to drive tier selection.

A. LLM query routing and cascading

  • FrugalGPT [1], Hybrid LLM [2], and RouteLLM [3] route by answer adequacy, estimated difficulty, or model strength, but do not model the physical device executing a local tier.These systems primarily route among cloud-hosted models or a strong–weak model pair.Source: A. LLM query routing and cascading · p. 2

B. Multi-tier device, edge, and cloud LLM inference

  • ConsRoute [4] and EACO-RAG [5] establish device–edge–cloud routing, but neither uses thermal state; their evaluations use laptop/workstation hardware or server simulation.HybridInfer instead bases routing on the thermal state of a real smartphone.Source: B. Multi-tier device, edge, and cloud… · p. 2

C. Thermal-aware inference and scheduling

Prior thermal-aware inference work generally adjusts execution on a fixed model or controls a different workload, while prior RL routers couple decisions through cost budgets. HybridInfer positions phone thermal state as a signal for choosing among LLM capability tiers.

C. Thermal-aware inference and scheduling

  • Prior thermal management adjusts frequency or task placement while keeping the model fixed, whereas thermal-aware offloading and datacenter scheduling address different workloads or system hierarchies.HybridInfer extends thermal-aware control to capability-tier choice, where thermal relief can come with a change in response quality.Source: C. Thermal-aware inference and… · pp. 2–3

D. Reinforcement learning for LLM routing

  • Router-R1 [9] and SeqRoute [10] use cost-aware or budget-aware sequential routing, whereas HybridInfer’s decisions couple through physical thermal state changed by tier choices.Local execution reduces future headroom; offloading lets the device cool.Source: D. Reinforcement learning for LLM… · p. 3

E. On-device LLM measurement

  • Mobile LLM measurement studies document throughput and thermal throttling; HybridInfer treats thermal signal as a control input and reports runtime instability as well as throughput loss.The measurement studies cited are [13] and [14].Source: E. On-device LLM measurement · p. 3

F. Position

  • HybridInfer’s claimed distinction is selecting among LLM tiers of differing capability using on-device thermal headroom, with an explicit quality–thermal tradeoff and real Android validation.The paper separates this from same-task offloading and frequency tuning.Source: F. Position · p. 3

III. METHOD

The method implements a three-tier LLM router on a Galaxy S25+ with an edge workstation and cloud service. It uses the phone’s Android thermal signal alongside query complexity to inform routing.

A. System and hardware

  • The system routes among a quantized Llama 3.2 3B phone model, a retrieval-augmented Llama 3.1 8B edge model, and cloud GPT-4o.The phone is a Samsung Galaxy S25+ with a Snapdragon 8 Elite and Adreno 830; per-query logging includes latency, thermal measures, battery current, and generated text.Source: A. System and hardware · p. 3

B. Thermal signal

  • Android’s getThermalHeadroom API provides a forecast where 0 is cool and 1.0 is the severe-throttle threshold, alongside a discrete thermal status.On the tested S25+, the API returned non-NaN values that varied with load.Source: B. Thermal signal · p. 3

C. Router state, actions, and baselines

HybridInfer represents routing conditions with thermal-headroom and query-complexity bins, and evaluates them on a balanced prompt workload using frozen reference answers and automatic quality metrics.

C. Router state, actions, and baselines

  • Five headroom bins and three complexity bins form 15 routing states, with three tier actions and seven fixed, heuristic, and learned routing conditions.The complexity score combines word count with a multi-step flag; the benchmark’s integrity gate checks that prompt labels match the router’s computed bins.Source: C. Router state, actions, and baselines · p. 3

D. Workload

  • The seeded benchmark contains 60 held-out test prompts and 150 training prompts, balanced across three complexity tiers and five technical domains.Prompts are unique, the test and training sets are disjoint, and output order interleaves complexity levels.Source: D. Workload · p. 3

E. Gold references and quality scoring

  • Quality is scored against 210 frozen Claude Opus 4.8 reference answers, using BERTScore-F1 as the primary metric and ROUGE-L as a secondary metric.Claude serves as an out-of-pipeline judge rather than matching any evaluated tier’s model family.Source: E. Gold references and quality scoring · pp. 3–4

F. Reinforcement-learning router

HybridInfer trains its Q-table offline from device-collected transitions, rewarding answer quality and local execution while penalizing latency, cost, and thermal pressure. The reported evaluation adapts the planned protocol to account for on-device instability and long-prompt failures.

F. Reinforcement-learning router

  • Device-collected transitions are scored against frozen gold answers, then used to train the Q-table offline; the phone loads the table and selects actions by argmax.The reported reference-at-training-only design keeps the gold model out of the deployed inference path.Source: F. Reinforcement-learning router · p. 4
  • The reward combines BERTScore-F1 and an on-device locality bonus against normalized latency and cost and a penalty that rises toward the SEVERE headroom threshold.Without the locality bonus, edge and cloud dominate in every state; the bonus favors on-device execution when thermally feasible, while the throttle penalty can override it.Source: F. Reinforcement-learning router · p. 4

G. Experimental protocol and the cooled-run extension

  • The planned protocol cools the device to AP-core temperature, collects transitions with systematic tier coverage, and evaluates seven conditions over 60 test prompts in three replications.It calls for cooldowns between conditions, heat-up runs, confidence intervals, throttle-violation rates, and paired Wilcoxon tests against heuristic baselines.Source: G. Experimental protocol and the… · p. 4
  • Because sustained always-on-device runs crash, C1 and C5 use one fresh process launch per prompt; long prompts still wedge, so those conditions report 40 short and medium prompts plus a wedge rate.The single-shot launches begin cool; 22 of 40 long prompts wedge beyond the 240 s cap, so those failures are not caused by accumulated heat.Source: G. Experimental protocol and the… · p. 4, Figure 2 · p. 5

IV. RESULTS

On-device inference is both slower under sustained load and unstable: repeated queries can crash the app, while long prompts can wedge even from a cool launch. Routing off hot states to edge is associated with more stable completion.

A. Per-tier behavior

  • On-device throughput falls from roughly 14 to 20 tokens/s when cool toward 4 to 13 tokens/s under sustained load, compared with roughly 18 tokens/s at edge and 90 to 97 tokens/s for cloud GPT-4o.A fresh on-device process also incurs an approximately 4.8 s TTFT for OpenCL kernel compilation.Source: A. Per-tier behavior · p. 5

B. Sustained and long-context on-device inference is unstable

  • The app crashed or was killed after roughly 4 to 12 consecutive on-device queries, and 22 of 40 long prompts failed to complete within 240 s, including fresh cool launches.Crash logs point to the MLC-LLM runtime and Adreno OpenCL driver; failures also occur when the device is cool, and long generations can hang during prefill.Source: B. Sustained and long-context… · p. 5
  • C6 completes every replication, while trained C7 completes most queries by offloading many; its policy keeps simple queries on-device in warm-but-feasible states and sends hot-state queries to edge.The policy figure marks states not visited during collection as “n/a.”Source: B. Sustained and long-context… · p. 5, Figure 3 · p. 5

C. The learned policy is thermal- and complexity-aware

The learned router achieved the highest quality among adaptive conditions, significantly outperforming both hand-tuned baselines at the lowest adaptive-router cost. Its policy varied with thermal headroom and query complexity, while on-device-only conditions were competitive in quality on short and medium prompts but slower and unable to serve long prompts.

C. The learned policy is thermal- and complexity-aware

  • With the locality bonus, the policy offloaded in hot H3 states, kept simple queries on-device when cool, and sent complex queries to edge or cloud.Without the bonus, the policy always offloaded.Source: C. The learned policy is thermal- and… · p. 5

D. Condition comparison

  • Table I reports held-out prompt results as means with 95% confidence intervals; C1 and C5 were measured one query per fresh launch on 40 short and medium prompts.The other listed conditions used three replications over all 60 prompts; the learned router's data was partial (n = 87).Source: D. Condition comparison · pp. 5–6, Table I · p. 6
  • C7 had the highest quality of any adaptive condition, significantly exceeding C4 (p = 0.011) and C6 (p = 0.0015), at the lowest adaptive-router cost (0.0017 USD/query).The comparisons used matched prompts and a paired Wilcoxon signed-rank test.Source: D. Condition comparison · pp. 5–6, Table I · p. 6
  • On the same 40 short and medium prompts, on-device conditions reached 0.8385 and 0.8403 BERTScore, while the learned router reached 0.8446 and ran about three times faster.On-device queries took 28–30 s, versus 9.0 s for the learned router; the router also served long prompts that wedged the device.Source: D. Condition comparison · p. 6

V. DISCUSSION

The learned router combines higher quality than hand-tuned heuristics with the lowest cost among adaptive conditions. It uses a state-dependent policy that keeps simple queries local when feasible and offloads complex or hot-state queries.

A. What the learned router buys

  • The router achieved the highest answer quality among adaptive conditions, significantly above both hand-tuned heuristics on matched prompts, at the lowest adaptive-router cost.It matched cloud-level quality while keeping some queries local and spending less than the heuristics.Source: A. What the learned router buys · p. 6
  • The policy keeps simple queries on-device when cool enough and offloads complex queries or queries in hot states to edge or cloud.It tunes thresholds from data against a reward the heuristics do not optimize.Source: A. What the learned router buys · p. 6
  • Rather than applying one fixed routing rule, the learned policy adapts its tier choice to query and device state.The section characterizes this as learning the heuristics' intuition while tuning thresholds from data.Source: A. What the learned router buys · p. 6

B. Locality has to be valued for thermal routing to mean anything

The paper argues that thermal-aware routing requires explicitly valuing on-device locality; otherwise the faster edge and cloud tiers attract every query. It also reports sustained on-device inference as a reliability and coverage hazard, with several limits on the measurements and device scope.

B. Locality has to be valued for thermal routing to mean anything

  • Without a locality bonus, optimizing quality, cost, and latency leads the policy to offload every query, making thermal state irrelevant to its decisions.The bonus credits privacy, offline capability, and zero marginal cost, making the device tier preferable when thermally feasible.Source: B. Locality has to be valued for… · p. 6

C. Thermal awareness as reliability, not only efficiency

  • Continuous always-on-device use crashes within a few queries, and a majority of long prompts wedge the runtime; conditions offloading when hot completed every query type.The paper frames load- and thermal-aware routing as a way to stay within a safe operating envelope, not only as a performance measure.Source: C. Thermal awareness as reliability,… · p. 6

D. Limitations

  • The on-device-heavy conditions were measured as cooled, single-query launches on short and medium prompts, so their tabulated values understate sustained-use throttling and instability.The learned router's three-replication data was also partial because warm-device crashes left n = 87.Source: D. Limitations · pp. 6–7

E. Future work

The paper identifies locality in the reward as necessary for thermal-aware routing to matter and sustained on-device inference as a reliability and coverage risk. It bounds these conclusions to a partial, single-device study and outlines experiments to test broader transfer and comparisons.

E. Future work

  • Future work includes completing the learned router's three replications, comparing against a reimplemented non-thermal three-tier router, and testing a harder, larger prompt workload.The current learned-router data is partial at n = 87.Source: E. Future work · p. 7

VI. CONCLUSION

  • The conclusion identifies explicit locality valuation as a precondition for meaningful thermal-aware routing, since otherwise the policy offloads every query.It also characterizes sustained on-device inference as a reliability and coverage hazard under the current toolchain.Source: §VI · p. 7
  • The reported results come from a single, uncooled, three-replication run, with partial learned-router data and single-shot measurements for on-device-heavy conditions.Proposed extensions include device-model evaluation and explicit privacy and energy reward terms.Source: §VI · p. 7

Related papers

Loading 2609.30270v1…