LLMs or Naive Bayes? Old Gems or New Ways

Mohammad Firas Sada; Dmitry Mishin; John Graham; Seungmin Kim; Mahidhar Tatineni; Frank Würthwein

Source-linked AI summary

LLMs or Naive Bayes? Old Gems or New Ways

Mohammad Firas Sada, Dmitry Mishin, John Graham, Seungmin Kim, Mahidhar Tatineni, Frank Würthwein

arXiv:2609.13185v1Jul 31, 2026cs.LGcs.AIcs.CLcs.DC

TL;DR

The paper examines whether Naive Bayes should be replaced by large language models for text classification under realistic resource constraints. It benchmarks Complement Naive Bayes against zero- and few-shot LLMs across tasks, scales, and operating conditions. LLMs lead mainly without labels, while Naive Bayes is usually competitive or superior with labels and substantially more efficient.

Problem
The paper asks whether classical Naive Bayes should be retired despite the accuracy and scale of modern LLMs, especially given HPC resource costs.
Method
The study benchmarks Complement Naive Bayes against zero- and few-shot LLMs across four model families, three classification tasks, training-size curves, throughput, energy, and an operational routing policy.
Results
LLMs dominate mainly in zero-data settings, whereas with labels Naive Bayes matches or beats zero-shot LLMs and is 40–486× faster, with roughly two orders of magnitude lower energy per sample.
Takeaways & Limitations
For resource-constrained HPC text classification, use Naive Bayes when labeled data exists and reserve LLMs for zero-data or ambiguous cases.
Takeaways & Limitations
The evaluation uses English benchmarks that favor bag-of-words, so specialized, multilingual, biomedical, long-context, or much larger-model settings may change the trade-off.

Abstract

from arXiv · show

Large language models (LLMs) prompt a recurring question in research computing: should classical methods like Naive Bayes (NB) be retired? We benchmark Complement Naive Bayes against zero-shot and few-shot LLMs spanning four model families and a 37x range in scale (27B to a 1T-parameter mixture-of-experts) across text classification tasks. LLMs dominate only in zero-data regimes (98.0% vs 88.2% on Amazon Polarity sentiment), and even that win is contamination-prone: on a low-contamination sentiment task NB beats the zero-shot LLM (81.7% vs 73.0%). However, once labeled data is available (e.g., AG News), NB reaches 89.1% accuracy, statistically indistinguishable from the zero-shot 27B LLM (89.0%) and better than the 397B frontier model (84.8%), at thousands of samples/sec on a commodity CPU. Fine-tuned DistilBERT reaches 90.6% but at far lower throughput than NB at batch size 1 (Table 2). Our measured GPU throughput analysis shows small-LLM batched inference is 40-486x slower than NB CPU inference (the multiplier depends strongly on the host CPU), exposing a structural gap bounded by memory bandwidth, with roughly two orders of magnitude lower energy per sample. For resource-constrained HPC practitioners performing text classification with labeled data, NB remains the optimal choice. We show the decision line is task-dependent (NB reaches LLM parity around $N \sim 10^4$ labels for topic classification, while zero-data sentiment favors the LLM at all N tested) and provide a Kubernetes Helm operator that automates model selection using configurable thresholds and verifiable Prometheus metrics.

1 Introduction

The paper asks whether classical Naive Bayes should be replaced by LLMs for large-scale text classification, emphasizing the resource costs of that choice. It benchmarks Complement Naive Bayes against zero- and few-shot LLMs and identifies when each approach is preferable.

Naive Bayes trains in under one second on a commodity CPU and classifies at sub-millisecond speeds per sample.These resource characteristics make it attractive for shared HPC environments facing energy and financial costs.
LLMs hold a structural advantage without labeled data, whereas labeled-data pipelines let Naive Bayes match or exceed zero-shot LLMs.The comparison spans models from 27B to 1T parameters and uses task-dependent decision boundaries.
40–486× throughput advantage for Naive Bayes over LLMs is measured across batch sizes and two accelerators.The gap is attributed to memory-bandwidth saturation and motivates cost-aware model selection.

2 Related Work

Prior work establishes Complement Naive Bayes and competitive TF-IDF linear models, while recent studies explore label-efficient and zero-shot alternatives. This paper instead locates the training-size point where Naive Bayes reaches LLM parity and examines energy costs of large-scale inference.

The study distinguishes its contribution by using training-size curves to locate where Naive Bayes reaches LLM parity.This directly characterizes the low-label regime rather than relying only on fixed zero-shot comparisons.
Prior methods include Complement Naive Bayes, TF-IDF linear models, SetFit, parameter-efficient few-shot tuning, and zero-shot LLM benchmarks.These approaches frame the paper’s comparison between classical classifiers and newer label-efficient methods.
Related HPC systems work finds that accelerators improve energy efficiency but do not match classical methods’ per-sample efficiency.The paper places its resource analysis within broader work on LLM serving and scientific infrastructure.

3 Methodology

The methodology compares CPU classical baselines, a fine-tuned transformer, and generative and discriminative LLMs across three stratified text-classification benchmarks. It measures predictive performance, throughput, power, and reproducibility under controlled Kubernetes-based experiments.

3.1 Models: The model suite includes CPU-only Complement Naive Bayes and Logistic Regression, 66M-parameter DistilBERT, and generative and discriminative Qwen models.Qwen3.6-27B is evaluated zero-shot and few-shot, while Qwen3.5-397B provides a frontier-scale comparison.
Three stratified benchmarks cover binary sentiment, four-class topic classification, and noisy 20-class topic classification.Amazon Polarity and AG News use 10K/2K train/test splits, while 20 Newsgroups uses 11,314/7,532 after removing headers and footers.
Accuracy, weighted F1, McNemar tests, paired-bootstrap confidence intervals, throughput, GPU power, and carbon footprint are measured.Throughput uses pre-tokenized samples across batch sizes 1–128 on RTX 3090 and Tesla V100 GPUs, with CPU baselines on dual Xeon Gold 6248R processors.
Accuracy measurements use managed vLLM endpoints, while controlled power measurements use self-hosted OLMo-2-1B and DistilBERT for attributable GPU-board power.Generation uses temperature 0 and a maximum sequence length of 8,192.
The batch-1 lower bound t_min = 2P/B_mem models weight-streaming time, and measured OLMo-2-1B throughput reaches about 63% of that bound.Batching beyond size 8 does not improve throughput because decoding remains weight-streaming-bound rather than compute-bound.
Experiments fix data splits, tokenizer settings, and vLLM flags, use namespace-scoped quotas, and provide a public artifact bundle for reproducing Tables 1–5.The runs execute as Kubernetes batch jobs on the NRP Nautilus cluster.

4 Results

Across tasks and model families, labeled data largely removes the LLM accuracy advantage, while Complement Naive Bayes retains a major efficiency advantage.

4.1 Amazon Polarity: Binary Sentiment: 98.0% few-shot Qwen3.6-27B accuracy on Amazon Polarity exceeds CNB’s 88.2%, but the result is flagged as contamination-prone.Amazon Polarity has heavy pre-training overlap, and the 397B model does not improve on the 27B model.
4.2 AG News: Clean Multi-Class: 89.1% CNB accuracy on AG News is statistically indistinguishable from zero-shot Qwen3.6-27B at 89.0%, while DistilBERT reaches 90.6%.CNB classifies at 0.03 ms/sample versus approximately 13 seconds for the zero-shot LLM.
4.3 20 Newsgroups: Noisy Multi-Class: 71.2% CNB accuracy on 20 Newsgroups is statistically indistinguishable from zero-shot Qwen3.6-27B at 71.8%, while the 397B model falls to 66.4%.DistilBERT trails CNB at 67.2%, whereas fine-tuned Qwen3-8B reaches 76.3% at 97 ms/sample.
4.4 Cross-Family and Scale Generalization: Across four model families and 27B–1T parameters, label-rich topic tasks neutralize the LLM advantage, and larger models do not help.
4.5 GPU Throughput Analysis: 40–486× throughput gaps favor NB over GPU inference, with the multiplier depending on host CPU and GPU throughput saturating by batch 8.The measured regime is consistent with memory-bandwidth limits from streaming full model weights from VRAM.
4.5 GPU Throughput Analysis: NB’s labeled break-even point is task-dependent: it reaches parity around N≈10^4 on topic tasks but never within N≤10^4 on Amazon sentiment.

5 Discussion

The decision boundary between CNB and LLMs depends on task, label availability, and accuracy tolerance rather than a universal sample threshold. Hybrid routing and a configurable Helm operator turn these measured trade-offs into deployable policies.

Decision boundary: N≈10^4 labels marks NB–LLM parity for topic tasks, while NB never reaches parity within N≤10^4 on Amazon sentiment.DistilBERT overtakes NB near N≈250 on two cleaner tasks but trails it throughout noisy 20 Newsgroups; the cost advantage remains unchanged with label count.
Efficiency: 40–486× throughput and roughly two orders of magnitude lower energy per sample keep NB economical whenever its accuracy is acceptable.The GPU gap is attributed to full-weight streaming from VRAM, whereas sparse TF-IDF fits in CPU cache; annual fleet-scale CO2e is not extrapolated.
Hybrid cascade: Escalating only NB’s low-confidence samples can exceed both standalone models on AG News, but adds cost on 20 Newsgroups where NB already leads.A 24% escalation rate reaches 89.6% on AG News, while 20 Newsgroups favors NB-only routing.
Operationalization: The Helm operator exposes task-specific thresholds and Prometheus metrics to route requests between CPU CNB and GPU LLM endpoints.Its policy uses settings such as minimum labeled examples, maximum latency, and target throughput; the reported deployment context includes shared infrastructure and conservative CPU–GPU comparisons.
Practical guidance: Zero-shot LLMs remain the recommended choice without labels, but labeled high-volume workloads should switch to CNB once task-specific parity is reached.The guidance favors fine-tuned transformers only when a 1–3 percentage-point gain justifies GPU use, and notes that batching does not close the NB gap.

6 Conclusion

For labeled text classification on resource-constrained HPC systems, Complement Naive Bayes remains the practical default because it matches or exceeds large-model performance with much higher efficiency. The paper operationalizes this task-dependent choice through a configurable Helm operator with Prometheus verification.

≈89% accuracy at thousands of samples/sec makes Naive Bayes the practical choice for labeled HPC text classification.It requires no GPU allocation, while the measured throughput gap versus GPU-accelerated transformer inference is 40–486×.
Naive Bayes reaches parity with large language models only after task-dependent amounts of labeled data are available.For topic classification, the reported parity point is around N∼10^4 labels.
The Helm operator automates model selection with configurable thresholds, Prometheus-based verification, and 7.2 μs routing overhead.
Using classical baselines on the NRP substrate frees scarce GPU and network capacity for workloads that genuinely require accelerators.