PaperlayerView exact revision on arXiv

Source-linked AI summary

FrugalSOT - Frugal Search Over the Models

Pradheep P, Yuvanesh S, Harish KB, Keerthan Saai Reddy S, Joshva Devadas T, Naveenkumar J, Hemalatha K

arXiv:2608.21621v1cs.LG

TL;DR

On-device LLM inference is constrained by hardware resources, latency, privacy, and compression trade-offs. FrugalSOT addresses these constraints by classifying prompt complexity and cascading requests across increasingly capable models, with escalation when relevance is insufficient. On a Raspberry Pi 5, it reduced inference time and resource use while preserving relevant output quality relative to the strongest model.

  • Problem

    On-device LLM inference faces memory, computation, latency, connectivity, privacy, and compression constraints, especially for real-time or sensitive applications.

    Source: B. Challenges in Deploying LLMs · p. 2

  • Method

    FrugalSOT classifies prompt complexity using length, named entities, and syntactic structure, then routes requests through progressively more capable models with relevance-based fallback escalation.

    Source: C. Dynamic Model Selection · p. 3, F. Final Output Generation · p. 5, §IV · p. 5

  • Results

    FrugalSOT achieved up to 60% reduction in inference time compared to single-model approaches while maintaining output quality through selective model escalation.

    Source: §IV · p. 5

  • Takeaways & Limitations

    The results support adaptive model selection as an efficient approach for high-quality NLP inference in low-resource edge environments.

    Source: §IV · p. 5, §V · p. 5

Abstract

from arXiv · show

In on-device NLP tasks, limited resources of embedded hardware, such as the Raspberry Pi 5, require efficient inference strategies. This paper introduces FrugalSOT (Frugal Search Over The Models), a resource-aware model selection architecture for on-device NLP inference. FrugalSOT estimates each request's complexity by extracting features such as prompt length, named entity density, and syntactic complexity. The request is first made to the least complex model that is likely to pass a relevance threshold. If the output of that model falls short of the threshold, the request is made to a more complex model. It is important to note that the relevance threshold undergoes continuous updates in the background. using past validation outcomes in an adaptation process using a low-pass filtering mechanism, thus imparting adaptation to changing input patterns. Experimental results achieved on a Raspberry Pi 5 show that FrugalSOT reduces average inference time and overall computational resource use to a significant extent compared to a single-model baseline approach, without compromising output relevance to the same extent as the most sophisticated model. These results confirm that adaptive model selection can enable efficient, high-quality natural language processing inference on limited devices.

I. INTRODUCTION

FrugalSOT addresses the resource, latency, privacy, and accuracy challenges of deploying LLMs on constrained edge hardware by selecting models according to request complexity.

  • FrugalSOT dynamically selects the best LLM for each request complexity and cascades from smaller on-device models to larger models when necessary.Source: §I · p. 1
  • Embedded hardware limits LLM deployment because large models require substantial memory and transformer self-attention has O(n²) sequence-length complexity.Source: B. Challenges in Deploying LLMs · p. 2
  • Cloud-based LLM inference introduces latency, bandwidth demands, connectivity requirements, and privacy risks for real-time or sensitive applications.Source: §I · p. 1, B. Challenges in Deploying LLMs · p. 2
  • Compression methods reduce resource use but introduce precision or performance trade-offs that complicate edge deployment.Source: B. Challenges in Deploying LLMs · p. 2
  • Edge platforms differ in compute, memory bandwidth, runtimes, energy limits, and thermal constraints, requiring hardware-specific optimization.Source: B. Challenges in Deploying LLMs · p. 2

C. Architectures for Resource-Constrained Environments

Prior work has pursued smaller, more efficient language-model architectures to reduce the computational burden of resource-constrained deployment while retaining performance.

  • ALBERT, MiniLM, and ELECTRA reduce model size and computation through parameter sharing and efficient pretraining methods.Source: C. Architectures for… · p. 2
  • Mistral and LLaMA 2 Chat use LoRA innovations and sparse attention patterns to target high performance with lower resource use.Source: C. Architectures for… · p. 2

D. Privacy and Decentralization in AI Systems

FrugalSOT keeps inference on the client device and classifies prompts using lightweight linguistic features to guide resource allocation across complexity levels.

Prompt Complexity Classification

C. Dynamic Model Selection

FrugalSOT uses available memory and prompt complexity to choose progressively capable models, with the Raspberry Pi 5 operating in the lightweight configuration.

  • Devices below 8 GB use TinyLlama for low-complexity, TinyDolphin for medium-complexity, Gemma2 2b for high-complexity, and Phi 2.7b as fallback.Source: C. Dynamic Model Selection · p. 3
  • Devices above 8 GB use LLaMA2 7b, Mistral 7b, LLaMA3.2 3b, and LLaMA2 13b for low, medium, high, and fallback roles, respectively.Source: C. Dynamic Model Selection · pp. 3–4
  • The Raspberry Pi 5’s 4/8 GB configurations operate in the lightweight category to maintain stable performance within memory constraints.Source: C. Dynamic Model Selection · p. 4
  • The hierarchy matches model capacity to query complexity and uses fallback escalation to reduce computational overhead while maintaining output quality.Source: C. Dynamic Model Selection · p. 4

D. Relevance Evaluation and Fallback Mechanism

FrugalSOT evaluates response relevance against complexity-specific thresholds and escalates requests through a model hierarchy when quality is insufficient. This fallback design reserves larger models for cases that smaller models cannot satisfy.

E. Adaptive Threshold

FrugalSOT updates relevance thresholds with an exponential moving average so they adapt gradually to new relevance scores and changing user-input patterns. The selected smoothing factor emphasizes stability while preserving responsiveness.

  • The adaptive threshold follows an exponential moving average that combines the previous threshold with a new relevance score.Source: E. Adaptive Threshold · p. 4
  • T_old is the previous threshold, T_new is the recalculated threshold, α controls the new score’s influence, and R is the model-generated relevance score.Source: E. Adaptive Threshold · p. 4
  • The system sets α = 0.2, assigning 20% weight to new observations and 80% to historical performance.Source: E. Adaptive Threshold · p. 4
  • Higher α values above 0.5 caused threshold oscillations, whereas values below 0.1 adapted too slowly to changing user patterns.Source: E. Adaptive Threshold · p. 4
  • Low-pass filtering prevents abrupt threshold changes that could trigger unnecessary escalations, while supporting gradual adaptation without computational overhead.Source: E. Adaptive Threshold · pp. 4–5

F. Final Output Generation

Once a response meets the relevance threshold, FrugalSOT returns it together with confidence and threshold-status metadata. The system tracks these outputs for later tuning and analysis.

IV. RESULTS

On a Raspberry Pi 5, FrugalSOT assigns models by query complexity and reports lower processing time with a small relevance reduction relative to Phi 2.7b. The results also report improved resource utilization for edge inference.

  • FrugalSOT classifies queries as low, mid, or high complexity using prompt length, named entities, and syntactic complexity.Source: §IV · p. 5
  • Lightweight TinyLlama and TinyDolphin handle low- and mid-complexity queries in approximately 100 and 80 seconds, respectively.Source: §IV · p. 5
  • Gemma2 2b and Phi 2.7b are selectively used for high-complexity queries, with execution times of 240–280 seconds and more contextual, accurate responses.Source: §IV · p. 5
  • FrugalSOT achieves a relevance score of 0.844 with an optimized inference time of 200 seconds, comparable to the best-performing individual models.Source: §IV · p. 5
  • The efficiency analysis reports optimized edge-device resource utilization and a balance of speed, power efficiency, and output quality.Source: §IV · p. 5

V. CONCLUSION

FrugalSOT is presented as an efficient, scalable, adaptable edge-inference framework for real-time and resource-constrained systems. Its design combines dynamic model choice, fallback handling, and adaptive thresholding.

  • FrugalSOT targets real-time inference on resource-constrained edge devices.Source: §V · p. 5
  • Dynamic model choice, fallback handling, and adaptive thresholding support efficient processing and relevant, high-quality outputs.The framework is also described as minimizing energy consumption.Source: §V · p. 5
  • Future work will investigate model tiling and hardware acceleration to improve edge-device efficiency and adaptability.Source: §V · p. 5

APPENDIX

The paper makes its experimental results, supplementary materials, and source code publicly available. The project website provides documentation, usage instructions, and codebase links for reproducibility and collaboration.

  • Experimental results, supplementary materials, and complete source code are publicly available.Source: APPENDIX · p. 6
  • These resources support reproducibility, transparency, and open collaboration for the research community.Source: APPENDIX · p. 6
  • The project website provides documentation, usage instructions, and links to the codebase.Source: APPENDIX · p. 6

Related papers

Loading 2608.21621v1…