Confidently Wrong, Silently So: Auditing Undetectable Failures of a Deployed On-Device Language Model

Shashwat Pandey; Satwik Pandey; Suresh Raghu

Source-linked AI summary

Confidently Wrong, Silently So: Auditing Undetectable Failures of a Deployed On-Device Language Model

Shashwat Pandey, Satwik Pandey, Suresh Raghu

arXiv:2608.23663v1Aug 24, 2026cs.SEcs.AI

TL;DR

Developer-accessible on-device language models are rarely audited independently, despite operating without server-side moderation. This paper audits one shippable configuration across calibration, false-premise confabulation, and benign-prompt refusal. It finds task-asymmetric miscalibration and no reliable user-visible cue distinguishing confident errors from correct outputs.

Problem
Developer-shippable on-device models operate without typical server-side safeguards, yet independent evidence usually concerns vendor-benchmarked rather than deployable configurations.
Method
The paper conducts a reproducible red-team reliability audit of the developer-accessible model across calibration, false-premise confabulation, benign-prompt refusal, and surface detectability.
Results
The model shows task-asymmetric miscalibration, while confident-correct and confident-wrong outputs are surface-indistinguishable at AUROC 0.55.
Takeaways & Limitations
Single-generation cues cannot reliably support oversight, whereas a tunable black-box consistency wrapper recovers reliability without model access.
Takeaways & Limitations
The reported rates come from one vendor’s deployed model and may not generalize to other on-device models.

Abstract

from arXiv · show

Aligning deployed language models requires knowing when their outputs can be trusted, yet on-device models now ship to hundreds of millions of devices with no server-side moderation, and the configuration developers can actually deploy is rarely audited independently. We present a reproducible reliability audit of the developer-accessible on-device foundation model, framed as an oversight question: can a user or a resource-constrained developer tell when the model is wrong? Red-teaming it on calibration, confident confabulation on false-premise questions, and over-refusal of benign prompts, we find a \emph{task-asymmetric miscalibration}: its guardrails fail in opposite directions across tasks (confabulating on 69\% of false premises while refusing 18\% of entirely benign inputs), atop a self-reported confidence that is saturated and non-discriminative (AUROC 0.47; ECE 70, worst among comparable small models). Crucially, confident-correct and confident-wrong outputs are \emph{surface-indistinguishable}: a classifier over 15 user-visible features separates them at AUROC only 0.55 (equivalence-confirmed), leaving no signal for oversight at inference time. No cheap single-generation signal flags these failures ($\le$0.68 AUROC), whereas a black-box consistency wrapper requiring no model access recovers reliability (confident confabulation 75\%$\to$3\%; selective accuracy 43\%$\to$83\%) at a tunable cost. We contribute a model-agnostic audit protocol, a surface-indistinguishability test, and released code and frozen evaluation items as reusable infrastructure for auditing deployed models.

1 Introduction

On-device language models operate in user-facing applications without server-side moderation, making reliability safety-relevant. This black-box audit finds task-asymmetric guardrails and saturated confidence, while showing that user-visible outputs lack signals distinguishing confident correctness from confident error.

Audit scope: The audit independently red-teamed the developer-accessible foundation model through its public API on commodity hardware across calibration, false-premise confabulation, and benign-input refusal.The audited configuration was the general base model exposed through the public framework, rather than a private task-specialized adapter.
Findings: 69% of false-premise questions elicited confident confabulation, while 18% of benign summarization inputs were refused, revealing guardrails miscalibrated in opposite directions.This task asymmetry was unchanged by the vendor’s permissive mode.
Findings: ∼99% self-reported confidence was non-discriminative, with AUROC 0.47 and expected calibration error 70, worst among comparable small models.The saturated confidence undermines confidence as a reliability signal.
Oversight and mitigation: Confident-correct and confident-wrong outputs carried no user-visible signal, while cheap single-generation signals failed and O(N) consistency provided a viable black-box oversight path.The study supports auditing deployable configurations and avoiding reliance on a single forward pass; it also releases code and frozen evaluation items.

2 Related Work

Prior work evaluates on-device models mainly through capability benchmarks or vendor-reported configurations, while reliability signals are largely developed for server-hosted models with rich traces. Existing research separately studies confident errors, red-teaming, over-refusal, and black-box consistency, but does not connect confident errors to users’ ability to detect them.

On-device models and their evaluation: On-device models are shipped on consumer hardware and exposed through vendor frameworks, but evaluation has centered on capability benchmarks and vendor-reported configurations (Apple 2025; Gemma Team, Google DeepMind 2025; Llama Team, AI @ Meta 2024).Independent reliability studies of small models exist (Karmakar and Chatterjee 2026), but target open checkpoints in the standard research stack.
Calibration and single-pass uncertainty: Calibration research examines verbalized confidence, trace length, self-evaluation, semantic entropy, and self-consistency, but these signals are validated mostly on server-hosted models with rich reasoning traces.The cited approaches range from cheap single-pass signals to more expensive sampling-based estimators.
Confident errors, red-teaming, and over-refusal: Confident hallucination, adversarial red-teaming, over-refusal, and black-box consistency methods are active areas, yet prior work does not test whether users can distinguish confident errors from confident-correct answers.This gap motivates evaluating confident errors through an explicit user-undetectability lens.

3 Audit Setup and Threat Model

The audit targets the developer-shippable on-device foundation model under a user-facing threat model where confident errors may lack server-side moderation or fallback detection. It evaluates frozen task sets against comparable small-model peers using deterministic grading and paired statistical tests.

Model under test and access boundary: The audit tests SystemLanguageModel.default, a roughly 3B-parameter general-purpose model exposed through FoundationModels to third-party developers on commodity hardware.This configuration is chosen because it is the model developers can actually ship, rather than a privately benchmarked task-specialized adapter.
Peers and judges: The main comparison uses greedy decoding for Gemma-3-4B-it, Llama-3.2-3B-instruct, and Ministral-3B, with additional sampled-decoding re-evaluation.Greedy decoding provides the lowest-variance hosted setting and most closely matches the on-device model’s near-deterministic guided decoding.
Tasks and item sets: Four frozen item sets cover factual QA, false-premise confabulation and abstention, and summarization groundedness and over-refusal.Factual QA uses TriviaQA and Global-MMLU; the false-premise set contains 110 unanswerable items and 150 answerable controls, while summarization contains 310 items.
Metrics and protocol: Models receive identical items and deterministic grading across calibration, AUROC, abstention, confabulation, groundedness, and selective accuracy or coverage metrics.Cross-model differences use paired tests, with 95% confidence intervals unless stated otherwise; confidence is defined as verbalized confidence of at least 80/100.
Threat model: The threat model treats confident factual errors or answers to questions the model should decline as harmful events for end users lacking server-side moderation or a fallback model.The audit asks whether these errors can be distinguished from correct outputs using signals available to users or resource-constrained developers.

4 Findings

The on-device model’s reliability guardrails are task-asymmetrically miscalibrated: it confidently answers false-premise questions while refusing benign inputs, driven by saturated, non-discriminative confidence.

4.1 Calibration: 98.8 mean verbalized confidence separates correct from incorrect factual answers at only AUROC 0.471, at or below chance.Confidence is similarly saturated under guided decoding (98.8) and free-form decoding (∼99), so the failure is not a decoding-mode artifact.
4 Findings: Across tasks, the model is worst-in-class on calibration, confident confabulation, and over-refusal, despite being most grounded on its intended task.These comparisons use identical items and one deterministic grader; peer serving-pipeline differences remain a confound.
4.2 False-premise questions: 69.1% of false-premise questions elicit confident confabulations, exceeding Gemma by 42.7 points, Llama by 56.4, and Ministral by 61.8 points.On 110 items, the model abstains only 30.9%, even though abstention or premise correction is appropriate.
4.3 Benign inputs: 86.2% of summarization items are grounded, yet the model refuses 18.4% of benign inputs while every peer refuses 0.0%.Apple’s permissiveContentTransformations guardrail mode leaves over-refusal essentially unchanged (18.8%→18.4%).

5 The Harm Mechanism: Surface Indistinguishability

Confident-wrong and confident-correct outputs are effectively indistinguishable from a single output’s user-visible surface features. Across 15 features, correctness prediction barely exceeds chance and is statistically indistinguishable from chance under equivalence testing.

Audit setup: On N=3019 high-confidence short-answer outputs, the audit compares 15 user-visible features that users or resource-constrained developers can read from a single output.The sample contains 858 correct and 2161 wrong outputs, including 108 false-premise confabulations.
Equivalence test: All 15 per-feature contrasts are δ-equivalent, and both classifiers’ upper 90% bounds fall below 0.60, confirming statistical indistinguishability from chance.Flesch readability survives Holm correction but has a negligible effect size of δ=−0.046.

6 Can Cheap Signals Flag the Confident Failures?

Cheap single-generation signals do not robustly identify confident failures across tasks: response length helps only for reasoning, while k-sample self-consistency is the only robust discriminator. SelfCheck improves AUROC over the strongest O(1) baseline on both factual and reasoning tasks, but at additional tunable sampling cost.

Single-generation signals: AUROC 0.513 versus 0.680: response length is the only cheap single-generation signal with power, but it is task-dependent and informative only on reasoning.The pattern is consistent with response length acting as a difficulty proxy that exists only when the model deliberates.
Single-generation signals: Three length-control tests show HVR adds nothing beyond response length, with partial-coefficient and likelihood-ratio p=0.69/0.98 and residualized HVR at chance.Length-residualized HVR is 0.449 [0.379, 0.519] and 0.502 [0.423, 0.582] across the two tasks.
Consistency sampling: Δ+0.194 and Δ+0.108: SelfCheck outperforms the strongest O(1) baseline on factual QA and reasoning, respectively, with significant DeLong tests.The factual-QA comparison is against fused confidence (DeLong p=1.3e−4), while the reasoning comparison is against response length (DeLong p=2.3e−3).

7 Mitigation: Black-Box Recovery Without Model Access

A black-box SelfCheck wrapper recovers reliability on the deployed on-device model by sampling multiple answers and abstaining when they disagree, without model access. At k=5, it sharply reduces confident errors and improves selective accuracy, with tunable cost and coverage.

7 Mitigation: Black-Box Recovery Without Model Access: SelfCheck draws k stochastic samples and abstains or escalates when final answers disagree, converting undetectable confident errors into explicit abstentions.The wrapper requires only black-box query access, with no logits, fine-tuning, or model internals, and runs on device.
7 Mitigation: Black-Box Recovery Without Model Access: 75.0%→2.6% confident confabulation and 42.7%→82.8% selective accuracy at 28.2% coverage demonstrate substantial recovery with k=5 and no model access.On factual QA, confident hallucination falls from 57.3% to 4.9%, while ECE improves from 57.2 to 17.2.
7 Mitigation: Black-Box Recovery Without Model Access: SelfCheck agreement separates correct from wrong answers at AUROC 0.77 on TriviaQA and 0.87 on GSM8K, while O(1) signals are not robust and task-general.The consistency signal beats the strongest cheap baseline on both tasks, with DeLong p < 0.003.
7 Mitigation: Black-Box Recovery Without Model Access: The wrapper costs k× generation, but k and the accept/defer threshold τ tune the accuracy–coverage tradeoff; k=5 captures most available signal.Independent samples permit on-device batching, keeping wall-clock latency below the nominal k× factor, while escalation can target high-stakes or low-agreement queries.

8 Discussion

The discussion identifies task-asymmetric guardrail failures and surface-indistinguishable errors as oversight failures, while exposing an accountability gap between benchmarked and deployable configurations. It argues that saturated confidence makes cheap single-generation uncertainty signals unreliable for deployment decisions.

Task-asymmetric miscalibration: Under-abstention on false premises and over-refusal on benign inputs persist in opposite directions even under the vendor’s permissive guardrail mode.A more permissive setting would worsen confabulation, whereas a more conservative setting would worsen over-refusal, making a single mis-set threshold insufficient as an explanation.
Why undetectability is the load-bearing finding: Across fifteen user-visible features, confident-correct and confident-wrong outputs are statistically equivalent, removing users’ ability to recognize errors as suspect.The authors identify surface indistinguishability as the load-bearing finding and report that errors are less surface-detectable than any peer’s.
The accountability gap: benchmarked is not shippable: The vendor-benchmarked configuration differs from the developer-deployable configuration, so private adapter capability and safety claims do not transfer to the general base model.Because the adapter is not exposed, third parties cannot independently audit the base model against those claims, creating an accountability gap.
Implications for deployment and evaluation: In this deployed on-device setting, saturated self-reported confidence, uninformative hedging, and conditionally informative length make cheap single-generation uncertainty signals unreliable.Length is informative only when the model genuinely reasons, so practitioners should not rely on a single forward pass for confidence assessment in models with confidence saturation.

9 Limitations

The audit evaluates one vendor’s deployed on-device model, so its reported rates may not generalize, even though the methodology, surface-indistinguishability protocol, and black-box wrapper are model-agnostic. The harm analysis establishes only the absence of user-visible cues, while API-based peer comparisons and a single mitigation operating point leave pipeline and budget effects unresolved.

Scope: The study audits a single deployed model from one vendor, limiting generalization of its specific rates despite model-agnostic methodology and mitigation components.The calibration, surface-indistinguishability, and black-box-wrapper methods are presented as model-agnostic, but the measured rates may not transfer to other on-device models.
Harm evidence: The harm argument establishes absence of a user-visible cue, not downstream harm rates, which would require a human-subjects study.
Deployment conditions: Peer models are served through hosted APIs rather than on device, so serving-pipeline differences cannot be fully excluded, and the mitigation is shown at only one tuned operating point.The mitigation is not swept across all deployment budgets.

10 Conclusion

The audit finds task-asymmetric miscalibration in a widely deployed on-device language model, combining confident confabulation with over-refusal and a saturated, non-discriminative confidence signal. Because confident errors are surface-indistinguishable from confident-correct outputs, users lack a cue to detect them, and no cheap single-generation signal recovers one.

Conclusion: The audit found task-asymmetric miscalibration: confident confabulation where caution is warranted and over-refusal where it is not.This result concerns the developer-accessible configuration of a widely deployed on-device language model.
Conclusion: A saturated and non-discriminative confidence signal underlies the model’s miscalibration.
Conclusion: Confident errors are surface-indistinguishable from confident-correct outputs, leaving users no cue to detect them and no cheap single-generation signal that recovers one.

Ethical Statement

The work is an independent, public-API reliability audit intended to improve oversight of deployed on-device models by documenting user-facing failures and providing a black-box mitigation.

Ethical Statement: The audit used a public API, public benchmark data, and no human subjects, reporting findings as measurements with confidence intervals rather than targeting any vendor.Its stated purpose is to document failures reaching users without server-side moderation and improve oversight through a black-box mitigation.