Source-linked AI summary
More Accurate or More Efficient? Evaluating Locally Deployed Compact Open-Weight Language Models for Mathematical Reasoning
Orion Powers, Daniella Seum, Khaled Slhoub
TL;DR
Local LLM evaluations often emphasize accuracy without measuring the runtime and energy costs that determine practical deployment. This paper introduces a controlled procedure for mathematical reasoning evaluation and applies it to three compact open-weight models. Accuracy and efficiency rankings diverge across datasets, with Gemma3:4b delivering roughly three times more correct answers per watt-hour than Qwen3:4b while Qwen3:4b leads accuracy on two datasets.
Problem
Existing evaluations often omit local runtime, token, energy, failure-mode, and paired statistical measurements, although these factors matter when models run on user-controlled hardware.
Method
The paper applies a controlled procedure with shared prompting, fixed settings, hierarchical answer extraction and verification, explicit failure classification, matched datasets, and per-question resource measurement.
Results
Accuracy and efficiency rankings disagree: Qwen3:4b leads accuracy on two datasets, Gemma3:4b leads Calculus I, and Gemma3:4b returns roughly three times more correct answers per watt-hour than Qwen3:4b on every dataset.
Takeaways & Limitations
Selecting a locally deployed model requires considering accuracy together with efficiency and intended use case.
Takeaways & Limitations
The study is limited by unavailable evaluation artifacts, unretained exact hardware and software identifiers, and possible dataset label noise or training-data contamination.
Abstract
from arXiv · showhide
Large language models are increasingly deployed on local hardware for privacy, cost, and accessibility reasons. Yet many evaluations emphasize accuracy while fewer quantify local runtime and energy, characterize failure modes, or apply paired statistical comparisons under controlled conditions. This paper presents a controlled, documented procedure for evaluating locally hosted LLMs on mathematical reasoning. It combines fixed inference settings, hierarchical answer extraction and verification, explicit failure-mode classification, and per-question resource measurement, and reports accuracy with paired significance tests and effect sizes. We demonstrate it in a preliminary study of three compact open-weight models under five billion parameters, Gemma3:4b (Google), Phi3:3.8b (Microsoft), and Qwen3:4b (Alibaba), across datasets spanning Grade 8 Math, Calculus I, and Advanced Probability and Statistics. All models ran through the same local inference server on one workstation, using a shared prompt template, controlled settings, and a matched question set per dataset. No single model dominates. Qwen3:4b is most accurate on two datasets and Gemma3:4b on Calculus I, yet Gemma3:4b returns roughly three times more correct answers per watt-hour than Qwen3:4b on every dataset while generating far fewer output tokens; Qwen3:4b requires substantially more generation time, energy, and output per question. Phi3:3.8b is substantially less accurate on all three datasets; its low extraction-failure rate indicates incorrect answers rather than unparsed output, though we caveat possible prompt-format effects. These preliminary findings indicate that accuracy alone is an insufficient basis for selecting a local model.
I. INTRODUCTION
The paper addresses gaps in evaluating locally hosted LLMs: comparisons often use inconsistent conditions, and accuracy alone omits practical resource costs. It proposes a controlled preliminary study that jointly measures correctness, efficiency, and failure modes.
- Independent model comparisons often differ in datasets, metrics, and experimental conditions, making reported results difficult to interpret.
- Local deployment makes runtime, generated tokens, and electricity direct operating considerations that can vary substantially between similarly sized models.
- The study evaluates Gemma3:4b, Phi3:3.8b, and Qwen3:4b across Grade 8 Math, Calculus I, and Advanced Probability and Statistics.All three models are compact open-weight systems with fewer than five billion parameters.
- The procedure standardizes prompting and inference, verifies extracted answers, records correctness and system-level metrics, and preserves question independence.
- Failure-mode classification separates wrong answers from extraction and execution failures, while paired tests and effect sizes compare accuracy under identical conditions.
- The study is preliminary because it covers three models on one hardware configuration in one task domain, with evaluation artifacts unavailable for release.
II. RELATED WORK
Prior work evaluates broad capabilities, mathematical reasoning, and inference efficiency, but this study positions its contribution as a narrower controlled procedure for local mathematical-model comparisons.
- MMLU, HELM, and BIG-Bench evaluate broad model capabilities across multitask knowledge and multiple behavioral dimensions.
- Mathematical benchmarks use objectively checkable answers, but varied notation and embedded final answers make direct string comparison insufficient.
- Efficiency research treats resource use as an evaluation criterion and measures costs such as power, latency, utilization, and memory bandwidth.
- Recent open-weight families make compact local deployment more accessible, including Microsoft Phi, Google Gemma, and Alibaba Qwen models.
- This study uses a narrower controlled sequence of dataset preparation, standardized execution, answer verification, and statistical metric collection.
A. Datasets
The evaluation uses curated mathematical datasets spanning school, introductory university, and advanced undergraduate domains, with structured records and matched cross-model sets.
- Datasets: 644 Grade 8 Math problems cover five Common Core categories: Expressions and Equations, Functions, Geometry, Statistics and Probability, and The Number System.
- Datasets: 374 Calculus I problems cover limits, derivatives, integrals, and applications typical of introductory university calculus.
- Dataset preparation: Dataset records contain problem text, expected answers, and metadata such as topics, directions, solutions, alternate solutions, and difficulty labels.
- Model and set selection: Matched evaluation sets support cross-model comparison, while the selected sub-5B models are intended for realistic local deployment by students, researchers, and smaller teams.
C. Prompt Design and Inference Configuration
All models receive the same problem prompt and fixed inference configuration, with correctness determined from extracted final answers rather than reasoning traces and trials isolated from one another.
- Prompt design: A single shared template supplies each model with the mathematical problem and requests a consistently formatted final answer.
- Prompt design: The prompt requires the response to end with `FINAL_ANSWER: <your answer>` on a new line.
- Answer evaluation: Evaluation uses the extracted final answer rather than the reasoning trace because detailed reasoning may still lead to an incorrect answer.
- Inference configuration: Inference parameters remain constant across models, including num_ctx=4096, num_predict=16384, temperature=0.3, top_p=0.9, and top_k=40.Qwen3:4b additionally uses `/no_think` to select its non-thinking mode.
- Trial control: Each query runs without conversational history, with a cooldown and temperature condition between models; a fixed warmup loads weights before each run.The warmup request is excluded from reported results.
D. Answer Extraction and Verification
The procedure extracts and verifies answers through hierarchical parsing and classifies outcomes to distinguish incorrect answers from extraction or execution failures.
- A multi-stage procedure handles inconsistent mathematical output formats, including reasoning, LaTeX, units, equivalent forms, and natural-language answers.It first searches for the prompted final-answer marker, then applies fallback pattern searches and normalization.
- Answers are classified as Correct, Wrong Answer, Extraction Failure, or Hard Failure based on matching, parseability, and inference success.This separates parsed but incorrect responses from unusable output and failed requests.
- Per-question records include correctness, generation time, prompt and output tokens, GPU utilization, temperature, and power draw.GPU energy is estimated by integrating observed power over inference duration and converting it to watt-hours.
- Matched problems support paired McNemar accuracy tests with Yates correction, Wilson 95% confidence intervals, and Holm correction across model pairs.Resource metrics are summarized descriptively because per-question measurements were not retained for release.
F. Experimental Setup
The study evaluates three compact models on matched mathematical datasets under a single local-workstation setup, reporting accuracy and paired statistical comparisons.
- The evaluation used one Windows 11 workstation with Ollama, full requested GPU offload, approximately 4.0 GB peak VRAM, 110 W peak GPU power, and 46 GB peak system RAM.GPU resource metrics were sampled at approximately 2 Hz during queries.
- Model artifacts were fixed after download, but model digests and quantization identifiers were not retained, so later default-tag versions may differ.Evaluations were conducted between April 6 and April 17, 2026.
- Qwen3:4b achieved the highest accuracy on Advanced Probability and Statistics (0.974) and Grade 8 Math (0.783), while Gemma3:4b led Calculus I (0.864).Model rankings differ across mathematical domains.
- All nine pairwise accuracy comparisons were statistically significant at α = 0.05 after McNemar testing with Holm correction.Comparisons involving Phi3:3.8b had large Cohen’s h values, whereas Gemma3:4b–Qwen3:4b differences were much smaller.
- Gemma3:4b led correct answers per watt-hour on every dataset despite being outscored on raw accuracy for two of three.Qwen3:4b generated roughly three to four times more output tokens per question than Gemma3:4b.
- Gemma3:4b was second in every Grade 8 Math Common Core category, while Phi3:3.8b reached only 0.091 on its strongest category.The five categories cover 643 of 644 problems; one uncategorized problem remains in overall accuracy.
B. Energy and Efficiency
Energy efficiency differs from raw accuracy across the compact models, with Gemma3:4b producing more correct answers per watt-hour and shorter outputs than Qwen3:4b.
- 6.53, 4.45, and 4.84 correct answers per watt-hour were achieved by Gemma3:4b on Advanced Probability and Statistics, Calculus I, and Grade 8 Math.Qwen3:4b achieved 1.96, 1.62, and 1.42 on the same datasets.
- 1,507, 1,485, and 1,518 output tokens per question were generated by Qwen3:4b, compared with 440, 502, and 361 for Gemma3:4b.The passage associates this output-length difference with the energy-efficiency gap.
C. Failure Modes
Failure analysis found negligible extraction failures and showed that accuracy and energy-efficiency rankings diverged, so model choice depends on the prioritized metric.
- C. Failure Modes: 0.0% extraction failures occurred for Gemma3:4b and Qwen3:4b across all datasets, while Phi3:3.8b reached 0.16% only on Grade 8 Math.No hard failures occurred, so most incorrect responses were parsed wrong answers rather than formatting failures.
- C. Failure Modes: 97.4% of Grade 8 Math responses were classified as wrong answers for Phi3:3.8b, indicating successfully parsed incorrect outputs.This distinguishes its low accuracy from an inability to extract answers.
- A. Accuracy and Efficiency Diverge: Accuracy and energy efficiency did not align among the three evaluated models, and the best-performing model depended on the prioritized metric.Local deployment also exposes users to hardware load, power usage, runtime, and query volume.
- A. Accuracy and Efficiency Diverge: Qwen3:4b was most resource-intensive across all three datasets, while resource-minimum rankings varied between Phi3:3.8b and Gemma3:4b.Phi3:3.8b had the lowest medians on Advanced Probability and Statistics and Grade 8 Math; Gemma3:4b did on Calculus I.
- A. Accuracy and Efficiency Diverge: Qwen3:4b led raw accuracy on Advanced Probability and Statistics and Grade 8 Math, whereas Gemma3:4b led Calculus I.For many-query workloads, cumulative energy and runtime may make Gemma3:4b more attractive, while accuracy remains dataset-dependent.
- A. Accuracy and Efficiency Diverge: Similar parameter counts did not yield similar accuracy, energy efficiency, output length, or failure patterns.The results therefore do not support choosing a model based only on accuracy or parameter count.
B. Interpreting the Phi3:3.8b Result
Phi3:3.8b performed substantially worse than the other models, but the result should be interpreted as specific to this evaluation procedure rather than as a general capability estimate.
- Observed result: 0.025 Grade 8 Math accuracy was substantially lower than the other two models’ performance.The paper notes that differing datasets, prompts, and evaluation conditions prevent direct comparison with Phi-3-mini’s GSM8K result.
- Failure interpretation: Negligible extraction failures indicate that answers were parsed from nearly all Phi3:3.8b outputs.The low extraction-failure rate does not establish that the prompt was well posed for this model.
- Failure interpretation: A mismatch between the shared prompt template and Phi3:3.8b’s expected chat format could have produced fluent, extractable, and incorrect responses.The study did not perform the transcript-level inspection needed to exclude this explanation.
- Scope of interpretation: April 2024 release age for Phi3:3.8b versus March and April 2025 releases for Gemma3:4b and Qwen3:4b is contextual, not an established explanation.Results therefore characterize the evaluated model instances and configurations rather than the broader model families.
C. The Cost of Disabling Deliberation
Disabling Qwen3:4b’s thinking mode made its configuration more comparable to the other models, but may have changed both accuracy and resource use; several measurement boundaries limit interpretation.
- Configuration choice: Disabling Qwen3:4b thinking mode prevented internal deliberation from consuming the token budget before the visible final answer.The choice was intended to make its response configuration more comparable to Gemma3:4b and Phi3:3.8b.
- Configuration choice: Thinking-mode status may affect Qwen3:4b accuracy and resource consumption because deliberation requires additional token generation.The study did not directly compare Qwen3:4b with thinking enabled and disabled.
- Measurement boundaries: GPU-only energy estimates exclude CPU, host memory, storage, and cooling, while approximately 2 Hz sampling adds proportionally more error for fast models.Applying the same procedure improves comparability but does not eliminate measurement error.
- Measurement boundaries: Runtime and energy measurements may depend on the single workstation’s drivers, thermal conditions, and operating-system scheduling.Absolute values should therefore be interpreted as characteristic of that workstation.
- Scope boundaries: The results apply most directly to sub-5B open-weight models running mathematical reasoning tasks through Ollama, not larger, proprietary, non-mathematical, or differently configured systems.Changing model tags, digests, quantization levels, or inference frameworks may produce different results.
VII. CONCLUSION AND FUTURE WORK
The preliminary comparison shows that local model selection cannot be reduced to accuracy: accuracy and resource-efficiency rankings diverged across the evaluated tasks. Future work targets Phi3:3.8b prompt-format effects, Qwen3:4b deliberation, and reproducible artifacts.
- Conclusion: Qwen3:4b had the highest accuracy on Advanced Probability and Statistics and Grade 8 Math, while Gemma3:4b led on Calculus I.Paired tests found statistically significant accuracy differences between all three models.
- Conclusion: Gemma3:4b returned roughly three times more correct answers per watt-hour than Qwen3:4b on every dataset.Qwen3:4b also generated roughly two-and-a-half to four times more output tokens and used roughly three to four times as much GPU energy per question, depending on dataset.
- Conclusion: Accuracy alone cannot fully describe the performance of similarly sized local models, which differed substantially in accuracy, efficiency, and domain-specific behavior.The preferred model therefore depends on the evaluation criterion and intended use case.
- Future work: Future work prioritizes transcript inspection for Phi3:3.8b, direct Qwen3:4b thinking-mode comparisons, and a publicly released reproducible implementation.The planned implementation would preserve versioned model identifiers, complete system specifications, and per-question results.