Source-linked AI summary
On-Device Named-Entity Recognition: A Deployability Study of Accuracy, Cost, Reliability, and Confidence
Vinay Kumar Chaganti
TL;DR
On-device NER raises linked questions about deployable models, annotation-free evaluation, and trustworthy confidence. The paper measures nine systems across three paradigms and datasets, and finds that compact encoders offer a deployability advantage even when a 4 B instruct LLM is competitive on accuracy.
Problem
Practitioners need evidence about which NER models are deployable, how to evaluate their own data without human annotation, and whether confidence can support abstention or escalation.
Source: §1 · p. 1Method
Results
Compact encoders offer zero malformed output, millisecond-to-second latency, and one-ninth to one-twenty-fourth the parameters, while a 4 B instruct LLM can match or beat them on clean and long-form text.
Source: §8 · p. 5, §10 · p. 6Takeaways & Limitations
An all-local confidence-gated cascade gives a modest, corpus-dependent improvement over cost-matched random routing, with a statistically supported gap reported only on RSS-News.
Source: §6.3 · p. 5RSS-News is a single English news domain, its human gold was silver-seeded, confidence analysis covers only GLiNER, and characterization omits model-aware difficulty measures.
Source: §9 · p. 6
Abstract
from arXiv · showhide
Named-entity recognition (NER) is increasingly wanted on-device (no API, low latency, data kept local). The practitioner's question is not the leaderboard but which model is deployable, how to evaluate it without human annotation, and whether its confidence can be trusted. We answer these jointly. We place nine systems across three paradigms and 13 M to 8 B parameters: a classical tagger (spaCy), bidirectional-encoder specialists (GLiNER, 166 to 460 M), and generative LLMs run locally (Qwen3-0.6B/1.7B/4B-Instruct, DeepSeek-R1-1.5B/8B), on three datasets of differing character, and report accuracy plus two axes the literature omits: latency and output validity. Because our corpus (RSS-News) had no gold, we built silver gold from a cross-family LLM judge panel, then measured its fidelity against benchmark gold and a full human re-validation of the corpus (strict F1 0.95, an upper bound since the human gold was silver-seeded); gold provenance flips the paradigm ranking, moving from LLM-authored silver to human gold raises every encoder and lowers every generative model. On accuracy alone a 4 B instruct LLM is competitive (it leads on clean newswire), so the encoder's case is deployability: it matches or slightly trails at one-ninth to one-twenty-fourth the size, at millisecond-to-second latency, with zero malformed output, while the smallest generative models emit up to 27% invalid output on long inputs, a failure fixed by scale, not output budget. We then characterize GLiNER's per-span confidence: it ranks correctness well (AUROC 0.76 to 0.86) but is overconfident (ECE 0.24 to 0.47, halved by temperature scaling); thresholding gives a small honest out-of-sample F1 gain; an all-local small-to-large cascade gives a modest, corpus-dependent gain over cost-matched random routing; and confidence tracks correctness but not novelty. Every number recomputes offline from per-span records.
1 INTRODUCTION
The paper frames on-device NER as a joint problem of choosing a deployable model, evaluating it without human annotation, and deciding whether model confidence is trustworthy. It measures accuracy, cost, reliability, annotation-free evaluation fidelity, and confidence together.
1 INTRODUCTION
- The study addresses three practitioner questions: which small model is deployable, how to evaluate local data without human annotation, and whether confidence can support abstention or escalation.It evaluates accuracy alongside latency and output reliability, measures the trustworthiness of annotation-free evaluation, and studies per-span confidence as a reliability tool.Source: §1 · p. 1
- The paper measures these topics together at the sub-1B on-device operating point, adding output validity, a measured fidelity bound for annotation-free NER gold, and an all-local uncertainty-gated cascade.The authors distinguish these measured contributions from established findings on compact encoders, capable LLMs, and NER confidence.Source: §1 · p. 1
- The contribution combines a deployability frontier across paradigms and sizes, a fidelity-tested annotation-free evaluation protocol, and a calibration, selective-prediction, and all-local cascade study.The frontier includes output validity as a first-class metric, and the study reports confidence intervals throughout.Source: §1 · p. 1
2 RELATED WORK
Prior work establishes compact encoders, LLM-based NER annotation, calibration, selective prediction, and uncertainty-based routing. This paper positions its contribution as measuring these strands together, including output validity, gold fidelity, and an all-local NER cascade.
Small-model and generative NER
- Prior work shows compact encoders can perform NER, while recent assessments find mid-size LLMs can tie strong encoders on CoNLL yet lose on specialized text and over-generate spans, [6].GLiNER frames NER as span-type matching in a bidirectional encoder, with entity knowledge distilled from LLMs in UniversalNER and NuNER,,.Source: §2 · pp. 1–2
Annotation-free evaluation
- LLM annotation, diverse-model panels, and alternative-annotator tests are established; this study differentiates itself with a measured strict-F1 fidelity bound and a NER ranking flip by gold provenance,, [7], [8], [9], [10], [11].The passage describes ranking shifts toward LLM-style systems in IR and documented self-preference bias [9], [10].Source: §2 · p. 2
Calibration, selective prediction, routing
- NER uncertainty work includes span-level calibration and overconfidence, while prior routing includes cloud-LLM mention routing and calibrated small-to-large sentence-level NLU cascades,, [17], [18].The paper places its all-local, encoder-to-encoder, span-gated NER cascade in the gap between those configurations.Source: §2 · p. 2
Difficulty and novelty
- Unseen entities are a longstanding source of NER difficulty,, while attribute-bucketed evaluation and model-aware difficulty measures are established characterization tools,.Dataset-level statistics are also used in prior characterization work.Source: §2 · p. 2
3 STUDY DESIGN
The study is framed as measurement across datasets, model paradigms, and deployability metrics. It uses fixed dataset samples and records predictions so reported results can be regenerated offline without re-inference.
3 STUDY DESIGN
- The design treats the work as measurement rather than leaderboard chasing.The paper frames each design choice as following from that stance.Source: §3 · p. 2
- The evaluation covers CoNLL-2003 clean newswire, WNUT-17 noisy user-generated text with emerging entities, and RSS-News, 93 long-form articles sampled from 24 RSS feeds.RSS-News documents are about 4,000 characters, and its gold is human-validated.Source: §3 · p. 2
- The system ladder spans spaCy, GLiNER small/medium/large, and locally run Qwen3 and DeepSeek-R1 models, with matched-size instruct/reasoning systems to isolate reasoning effects.The models run on the same Apple M4 Pro hardware, and none is fine-tuned.Source: §3 · p. 2
- Accuracy is SemEval-strict F1 with document-bootstrap confidence intervals; the study also measures per-document latency and output validity, then regenerates tables from provenance-stamped span records.For schema-constrained generation, invalid output means failure to terminate a schema-valid object within the token budget.Source: §3 · p. 2
4 THE DEPLOYABILITY FRONTIER
Accuracy varies across dataset conditions, while output validity and latency distinguish deployable systems from some generative models. Per-type results also show that fixed schemas impose structural limits.
4.1 Accuracy: on clean text the gap is closed; encoders lead on noise
- Qwen3-4B-Instruct leads CoNLL (0.753) and RSS-News (0.684), while GLiNER-large leads WNUT (0.599).GLiNER-small (166 M) beats GLiNER-medium (209 M) on WNUT by 0.093; DeepSeek-R1-1.5B scores 0.398/0.264 on CoNLL/WNUT versus Qwen3-1.7B at 0.652/0.485.Source: §4.1 · p. 2, Table 2 · p. 3
4.2 Reliability and latency: where the generative arm is disqualified
- 27% of long RSS-News outputs from Qwen3-0.6B are invalid, versus 0% for Qwen3-4B-Instruct; raising the smaller model's token budget barely helps.Encoders run in 0.1 to 4 s, while DeepSeek-R1-8B takes 28 s per document; the reported validity failure is non-termination, not malformed decoding.Source: Table 3 · p. 3, §4.2 · p. 3
4.3 Per-type and significance
- PERSON is easiest and PRODUCT hardest; spaCy scores near zero on PRODUCT because its fixed schema lacks a slot, while bootstrap tests confirm the GLiNER-small-over-medium WNUT gap and tiny-LLM deficits.The passage characterizes the classical tagger's PRODUCT limit as structural rather than statistical.Source: §4.3 · p. 3
5 ANNOTATION-FREE EVALUATION AND ITS FI-
A cross-family judge panel supplies silver gold for RSS-News, and the authors measure its fidelity against benchmark and human gold. Changing from silver to human gold shifts the encoder-versus-generative ranking, though silver-seeded human validation makes that shift conservative.
5.1 A measured fidelity bound
- The cross-family judge panel labels spans by ≥2-of-3 consensus, using a model family different from every system under test.For RSS-News, the one-time cloud panel is an evaluation oracle, not part of the deployed system.Source: §5.1 · p. 3
- Against human gold, the panel reaches strict F1 0.952 on RSS-News, an upper bound because annotators edited silver-seeded output.The panel’s precision is 0.987 and recall is 0.919; human review added 143 missed entities and deleted 22 silver spans.Source: §5.1 · p. 3, Table 6 · p. 4
5.2 Gold provenance flips the ranking
- Moving from silver to human gold raises every encoder by up to +0.037 and lowers every generative model by down to −0.029, with GLiNER-large overtaking R1-8B.Because the human gold was seeded from silver, the observed flip is a conservative estimate of the provenance effect.Source: §5.2 · pp. 3–4, Table 7 · p. 4
6 CONFIDENCE: CALIBRATION, SELECTIVE PREDICTION, ROUTING
GLiNER’s native per-span confidence ranks correctness but is overconfident. The paper tests its use for abstention and all-local routing, and finds that it tracks correctness rather than novelty.
6.1 Ranks correctness, but miscalibrated
- GLiNER confidence separates correct from incorrect spans (AUROC 0.76 to 0.86) but is overconfident (ECE 0.24 to 0.47); temperature scaling roughly halves ECE.On RSS-News, temperature scaling changes ECE from 0.45 to 0.28.Source: §6.1 · p. 4, Figure 3 · p. 4, Table 8 · p. 5
6.2 Selective prediction, honestly measured
- Validation-transferred confidence thresholding raises strict F1 by +0.133 on WNUT and +0.143 on RSS-News for GLiNER-medium.Thresholds are selected on validation splits and applied to disjoint test splits; gains are averaged over 40 random splits.Source: §6.2 · pp. 4–5, Figure 4 · p. 5
6.4 Confidence tracks correctness, not novelty
- GLiNER-small’s mean confidence correlates with strict F1 but not unseen-entity ratio on CoNLL and WNUT.The reported Spearman correlations with strict F1 are +0.30 on CoNLL and +0.46 on WNUT; the novelty correlations are +0.07 and −0.05, both non-significant.Source: §6.4 · p. 5
6.3 An all-local confidence-gated cascade
- At 50% routed, the all-local cascade improves strict F1 over cost-matched random routing by +0.021 on RSS-News, while benchmark gaps remain within noise.The cascade routes the most-uncertain documents from GLiNER-small to GLiNER-large.Source: Figure 5 · p. 5, §6.3 · p. 5
7 WHAT MAKES IT HARD: A CHARACTERIZATION CAVEAT
Static per-document characteristics do not reliably predict NER difficulty within a dataset; only cross-dataset entity novelty tracks the ordering, a known and caveated result.
7 WHAT MAKES IT HARD: A CHARACTERIZATION CAVEAT
- Within-dataset correlations between static document characteristics and F1 are weak and dataset-specific (|ρ| < 0.17), while entity novelty tracks cross-dataset ordering.CoNLL test entities are 46% unseen in training versus 98% for WNUT; the study did not use model-aware per-instance difficulty measures.Source: §7 · p. 5
8 DISCUSSION
The discussion favors encoders for deployment on reliability, latency, and size rather than accuracy alone, while treating GLiNER confidence as a ranking signal and examining selective prediction and routing.
8 DISCUSSION
- A fair 4 B instruct LLM can match or beat encoders on clean and long-form text, while encoders offer 0% malformed output, millisecond-to-second latency, and 1/9 to 1/24 the parameters.GLiNER-large is the accuracy-leaning on-device choice, while GLiNER-small is the efficiency default.Source: §8 · pp. 5–6
- Table 9 compares keep-all strict F1 with validation-chosen thresholds transferred to test, reporting mean and standard deviation over 40 splits.Source: Table 9 · p. 6
- Table 10 reports endpoint F1, gate and random routing at 25% and 50% routed, and the routed fraction needed to reach 95% of the full gain.Source: Table 10 · p. 6
- Table 11 reports Spearman correlations between GLiNER-small mean confidence and document strict F1, and between confidence and unseen-entity ratio.Source: Table 11 · p. 6
- Table 12 presents dataset characterization alongside unseen-entity ratio and model differentiation.Source: Table 12 · p. 6
- GLiNER confidence is overconfident as a probability (ECE up to 0.47), but ranks spans well enough for abstention and an all-local cascade; temperature scaling is the fix when probabilities are needed.Source: §8 · p. 6
9 LIMITATIONS
The study’s scope is bounded by its single-domain corpus and seeded human gold, GLiNER-only confidence analysis, limited novelty comparison, and lack of fine-tuning.
9 LIMITATIONS
- The study uses English news-domain RSS-News with human gold seeded from silver, analyzes confidence only for GLiNER, and compares novelty across two benchmark points without model-aware difficulty measures.No system is fine-tuned; a distilled on-device student is left for future work.Source: §9 · p. 6
10 CONCLUSION
The conclusion recommends compact bidirectional encoders for sub-1B on-device NER, pairing deployability with annotation-free evaluation and confidence-guided selection and routing.
10 CONCLUSION
- For sub-1B on-device NER, compact bidirectional encoders offer competitive accuracy with zero malformed output, low latency, and a fraction of generative models’ parameters.Their invalid-output advantage over small generative models is a scale property, not a tuning artifact; panel silver gold fidelity is 0.95 on RSS-News, an upper bound because human gold was silver-seeded.Source: §10 · p. 6