Source-linked AI summary
Task-Specific Efficiency Analysis: When Small Language Models Outperform Large Language Models
Jinghan Cao, Yu Ma, Xinjin Li, Qingyang Ren, Xiangyun Chen
TL;DR
Large models offer strong capabilities but impose computational costs that complicate deployment in resource-constrained environments, while systematic task-specific efficiency comparisons have been limited. This paper evaluates 16 models across five NLP tasks using the Performance-Efficiency Ratio, finding that 0.5–3B models achieve superior efficiency-performance trade-offs.
Problem
Existing studies lack systematic, task-specific efficiency comparisons using consistent evaluation protocols, despite the deployment costs of large models in resource-limited environments.
Method
The paper evaluates models from 0.5B to 72B parameters across five NLP tasks and introduces PER, which combines accuracy, throughput, memory usage, and latency through geometric-mean normalization.
Results
0.5–3B models achieve superior PER over larger counterparts, while models above 14B show diminishing returns.
Takeaways & Limitations
The findings provide a quantitative basis for task-specific model selection in resource-constrained deployments and challenge the “bigger is better” paradigm.
Takeaways & Limitations
Min-max normalization is sensitive to outliers, although the study uses a fixed set of 16 representative models and focuses on relative rankings.
Abstract
from arXiv · showhide
Large Language Models achieve remarkable performance but incur substantial computational costs unsuitable for resource-constrained deployments. This paper presents the first comprehensive task-specific efficiency analysis comparing 16 language models across five diverse NLP tasks. We introduce the Performance-Efficiency Ratio (PER), a novel metric integrating accuracy, throughput, memory, and latency through geometric mean normalization. Our systematic evaluation reveals that small models (0.5--3B parameters) achieve superior PER scores across all given tasks. These findings establish quantitative foundations for deploying small models in production environments prioritizing inference efficiency over marginal accuracy gains.
1 Introduction
Large models deliver strong capabilities but impose substantial computational costs, creating deployment challenges in resource-constrained settings. This paper addresses the lack of systematic task-specific efficiency comparisons by evaluating models across tasks and introducing a multi-dimensional efficiency metric.
- Large Language Models achieve broad capabilities but impose substantial computational costs that challenge deployment on resource-limited environments, edge devices, and real-time applications.
- The study addresses missing systematic, task-specific efficiency comparisons conducted under consistent evaluation protocols.
- The paper evaluates language models ranging from 0.5B to 72B parameters across five diverse tasks.
- The study introduces a unified evaluation framework and the Performance-Efficiency Ratio to capture multi-dimensional trade-offs.
2 Related Work
Prior work shows that small models can be competitive on selected tasks, but existing studies often emphasize capability or performance without systematic efficiency evaluation. This motivates broader comparisons across tasks and model scales.
- Related work spans evaluations of language-model reasoning, trustworthiness, domain boundaries, and societal impacts across specific application fields.
- Prior research has evaluated compact models for code generation and found competitive performance with substantially lower memory requirements.
- Pattern-exploiting training enabled small models to achieve GPT-3-comparable accuracy on eight SuperGLUE NLU tasks.
- Existing studies described in this review did not systematically combine performance evaluation with system-level efficiency comparisons.
3 Methodology
The methodology compares representative open-source models across five NLP benchmarks using task-appropriate efficiency measures and a Performance-Efficiency Ratio. PER combines normalized accuracy, throughput, memory, and latency through a geometric mean designed to expose imbalanced performance.
- Experimental Design Framework: The evaluation compares 16 open-source language models spanning 0.5B to 72B parameters across architectures.
- Experimental Design Framework: Five benchmarks cover sentiment classification, commonsense reasoning, scientific knowledge, and reading comprehension across different task complexities and output characteristics.
- Performance-Efficiency Ratio: PER combines accuracy, throughput, memory usage, and latency using a geometric mean after min-max normalization to [0, 1].
- Mathematical Formulation: Min-max normalization maps each component to [0, 1], using higher-is-better or lower-is-better transformations as appropriate.
- Mathematical Formulation: The geometric mean limits compensation across dimensions, so weakness on one metric can drive a model’s PER near zero.
- Mathematical Formulation: The fixed 16-model comparison and focus on relative ranking motivate min-max normalization despite its sensitivity to outliers.
- Task-Specific Metric Definitions: The analysis uses token-level throughput for generation and sample-level throughput for classification.
4 Experiment and Result
Across the evaluated tasks, scaling produces task-dependent accuracy gains, while efficiency results favor compact models. Small models achieve the strongest PER rankings in both generation and classification, despite some accuracy advantages for larger models.
- Experimental Setup: Experiments used NVIDIA A10 and A100 GPUs with tensor parallelism ranging from one to eight GPUs according to model size.
- Task-Specific Scaling Characteristics: Three scaling regimes appear: saturation for simple classification, pronounced but non-monotonic gains for mathematical reasoning, and diminishing returns for scientific and commonsense reasoning.
- Task-Specific Scaling Characteristics: 91.7% accuracy is reached by Qwen2.5-0.5B on IMDB, compared with 88.6% for Qwen2.5-72B, showing saturation at sub-billion scale.
- Generation Task Efficiency: 1.5–3B models achieve PER scores of 0.60–0.72 on generation tasks, while models above 14B remain below 0.35.
- Generation Task Efficiency: 350–4400 tokens/s/GPU gives compact generation models 40–90× higher throughput than larger alternatives at 20–50 tokens/s/GPU.
- Classification Task Efficiency: 0.5–2B models dominate classification PER rankings, with efficiency gains substantially exceeding accuracy trade-offs even on complex tasks.
5 Conclusion
Small models (0.5–3B) achieve the highest Performance-Efficiency Ratios, while larger models show diminishing returns beyond 14B parameters. Medium-scale models balance performance, but small models are most effective for resource-constrained deployments.
- Small models (0.5–3B) achieve superior Performance-Efficiency Ratios over larger counterparts.
- Larger models exhibit diminishing returns beyond 14B parameters.
- Medium-scale models (3–14B) offer an optimal performance balance.
- The findings provide a quantitative basis for task-specific model selection and challenge the “bigger is better” paradigm.