Source-linked AI summary
FrugalSOT - Frugal Search Over the Models
Pradheep P, Yuvanesh S, Harish KB, Keerthan Saai Reddy S, Joshva Devadas T, Naveenkumar J, Hemalatha K
TL;DR
On-device LLM inference is constrained by hardware resources, latency, privacy, and compression trade-offs. FrugalSOT addresses these constraints by classifying prompt complexity and cascading requests across increasingly capable models, with escalation when relevance is insufficient. On a Raspberry Pi 5, it reduced inference time and resource use while preserving relevant output quality relative to the strongest model.
Problem
On-device LLM inference faces memory, computation, latency, connectivity, privacy, and compression constraints, especially for real-time or sensitive applications.
Method
FrugalSOT classifies prompt complexity using length, named entities, and syntactic structure, then routes requests through progressively more capable models with relevance-based fallback escalation.
Results
FrugalSOT achieved up to 60% reduction in inference time compared to single-model approaches while maintaining output quality through selective model escalation.
Takeaways & Limitations
The results support adaptive model selection as an efficient approach for high-quality NLP inference in low-resource edge environments.
Abstract
from arXiv · showhide
In on-device NLP tasks, limited resources of embedded hardware, such as the Raspberry Pi 5, require efficient inference strategies. This paper introduces FrugalSOT (Frugal Search Over The Models), a resource-aware model selection architecture for on-device NLP inference. FrugalSOT estimates each request's complexity by extracting features such as prompt length, named entity density, and syntactic complexity. The request is first made to the least complex model that is likely to pass a relevance threshold. If the output of that model falls short of the threshold, the request is made to a more complex model. It is important to note that the relevance threshold undergoes continuous updates in the background. using past validation outcomes in an adaptation process using a low-pass filtering mechanism, thus imparting adaptation to changing input patterns. Experimental results achieved on a Raspberry Pi 5 show that FrugalSOT reduces average inference time and overall computational resource use to a significant extent compared to a single-model baseline approach, without compromising output relevance to the same extent as the most sophisticated model. These results confirm that adaptive model selection can enable efficient, high-quality natural language processing inference on limited devices.
I. INTRODUCTION
FrugalSOT addresses the resource, latency, privacy, and accuracy challenges of deploying LLMs on constrained edge hardware by selecting models according to request complexity.
- FrugalSOT dynamically selects the best LLM for each request complexity and cascades from smaller on-device models to larger models when necessary.
- Embedded hardware limits LLM deployment because large models require substantial memory and transformer self-attention has O(n²) sequence-length complexity.
- Cloud-based LLM inference introduces latency, bandwidth demands, connectivity requirements, and privacy risks for real-time or sensitive applications.
- Compression methods reduce resource use but introduce precision or performance trade-offs that complicate edge deployment.
- Edge platforms differ in compute, memory bandwidth, runtimes, energy limits, and thermal constraints, requiring hardware-specific optimization.
C. Architectures for Resource-Constrained Environments
Prior work has pursued smaller, more efficient language-model architectures to reduce the computational burden of resource-constrained deployment while retaining performance.
- ALBERT, MiniLM, and ELECTRA reduce model size and computation through parameter sharing and efficient pretraining methods.
- Mistral and LLaMA 2 Chat use LoRA innovations and sparse attention patterns to target high performance with lower resource use.
D. Privacy and Decentralization in AI Systems
FrugalSOT keeps inference on the client device and classifies prompts using lightweight linguistic features to guide resource allocation across complexity levels.
- On-device processing reduces cloud dependence and is presented as beneficial for privacy, latency, standalone operation, and offline or high-security environments.
- Prompt Complexity Classification: The classifier uses prompt length, named entity recognition, and syntactic structure in a weighted scoring model to assign low, mid, or high complexity.
- Prompt Complexity Classification: Length complexity is categorized as low for L ≤5, mid for 6 ≤L ≤10, and high for L > 10.
- Prompt Complexity Classification: NER complexity is low when Ce = 0, mid for 1 ≤Ce ≤3, and high when Ce > 3.
- Prompt Complexity Classification: Syntactic complexity is low for S = 0, mid for 1 ≤S ≤2, and high for S > 2, using conjunctions, subordinate clauses, and sentence length.
- Prompt Complexity Classification: The weighted score is C = 1 · LCscore + 2 · NERCscore + 3 · SCscore, with low complexity at C ≤4, mid at 5 ≤C ≤8, and high at C ≥9.
C. Dynamic Model Selection
FrugalSOT uses available memory and prompt complexity to choose progressively capable models, with the Raspberry Pi 5 operating in the lightweight configuration.
- Devices below 8 GB use TinyLlama for low-complexity, TinyDolphin for medium-complexity, Gemma2 2b for high-complexity, and Phi 2.7b as fallback.
- Devices above 8 GB use LLaMA2 7b, Mistral 7b, LLaMA3.2 3b, and LLaMA2 13b for low, medium, high, and fallback roles, respectively.
- The Raspberry Pi 5’s 4/8 GB configurations operate in the lightweight category to maintain stable performance within memory constraints.
- The hierarchy matches model capacity to query complexity and uses fallback escalation to reduce computational overhead while maintaining output quality.
D. Relevance Evaluation and Fallback Mechanism
FrugalSOT evaluates response relevance against complexity-specific thresholds and escalates requests through a model hierarchy when quality is insufficient. This fallback design reserves larger models for cases that smaller models cannot satisfy.
- Relevance is assessed using cosine similarity to all-MiniLM-L6v2, with thresholds derived from 250 prompt-response pairs across three complexity levels.The dataset contains 100 low-complexity, 100 medium-complexity, and 50 high-complexity pairs.
- When a model’s similarity score falls below its complexity threshold, FrugalSOT automatically escalates the request to the next model in the hierarchy.
- A TinyLlama score below 0.4441 triggers escalation to TinyDolphin, then Gemma2 2b, and finally Phi 2.7b if quality remains unsatisfied.
- Hierarchical fallback conserves computation and power by invoking higher-tier models only when lower-tier outputs fail the relevance requirement.
E. Adaptive Threshold
FrugalSOT updates relevance thresholds with an exponential moving average so they adapt gradually to new relevance scores and changing user-input patterns. The selected smoothing factor emphasizes stability while preserving responsiveness.
- The adaptive threshold follows an exponential moving average that combines the previous threshold with a new relevance score.
- T_old is the previous threshold, T_new is the recalculated threshold, α controls the new score’s influence, and R is the model-generated relevance score.
- The system sets α = 0.2, assigning 20% weight to new observations and 80% to historical performance.
- Higher α values above 0.5 caused threshold oscillations, whereas values below 0.1 adapted too slowly to changing user patterns.
- Low-pass filtering prevents abrupt threshold changes that could trigger unnecessary escalations, while supporting gradual adaptation without computational overhead.
F. Final Output Generation
Once a response meets the relevance threshold, FrugalSOT returns it together with confidence and threshold-status metadata. The system tracks these outputs for later tuning and analysis.
- FrugalSOT produces the final output when a response meets the threshold from either the base model or an escalated model.
- The output includes the primary response, a confidence score, and a marker for the current adaptive-threshold status.
- Outputs and associated metadata are tracked for future tuning and analysis.
IV. RESULTS
On a Raspberry Pi 5, FrugalSOT assigns models by query complexity and reports lower processing time with a small relevance reduction relative to Phi 2.7b. The results also report improved resource utilization for edge inference.
- FrugalSOT classifies queries as low, mid, or high complexity using prompt length, named entities, and syntactic complexity.
- Lightweight TinyLlama and TinyDolphin handle low- and mid-complexity queries in approximately 100 and 80 seconds, respectively.
- Gemma2 2b and Phi 2.7b are selectively used for high-complexity queries, with execution times of 240–280 seconds and more contextual, accurate responses.
- FrugalSOT achieves a relevance score of 0.844 with an optimized inference time of 200 seconds, comparable to the best-performing individual models.
- The efficiency analysis reports optimized edge-device resource utilization and a balance of speed, power efficiency, and output quality.
V. CONCLUSION
FrugalSOT is presented as an efficient, scalable, adaptable edge-inference framework for real-time and resource-constrained systems. Its design combines dynamic model choice, fallback handling, and adaptive thresholding.
- FrugalSOT targets real-time inference on resource-constrained edge devices.
- Dynamic model choice, fallback handling, and adaptive thresholding support efficient processing and relevant, high-quality outputs.The framework is also described as minimizing energy consumption.
- Future work will investigate model tiling and hardware acceleration to improve edge-device efficiency and adaptability.
APPENDIX
The paper makes its experimental results, supplementary materials, and source code publicly available. The project website provides documentation, usage instructions, and codebase links for reproducibility and collaboration.
- Experimental results, supplementary materials, and complete source code are publicly available.
- These resources support reproducibility, transparency, and open collaboration for the research community.
- The project website provides documentation, usage instructions, and links to the codebase.