Single-Query Black-Box Calibration Auditing via Logit Bias

• Calibration metrics such as Expected Calibration Error (ECE) [6] require continuous probabilities, which commercial APIs frequently hide and thereby limit independent auditing.LLMs used as zero-shot or few-shot classifiers commonly rely on target-token probabilities, such as “True” versus “False” [9].Source: §1 · p. 1
• Manipulating the logit_bias parameter lets the method evaluate exact probability thresholds and compute dataset calibration error without extracting continuous logits.Some major APIs, such as OpenAI, expose logit_bias, originally intended to control generation.Source: §1 · p. 1
• The proposed black-box calibration estimator requires one API query per sample and is proved asymptotically consistent to True Calibration Error (TCE).The paper decomposes the estimator’s bias and variance.Source: §1 · p. 1

2 Related Work

• Traditional calibration evaluation requires exact output distributions, and proxy methods can have severe cost or accuracy limitations.The proposed single-query estimator is presented as resolving these deficiencies.Source: §2 · p. 2

Proxy Baselines: Verbalization and Sampling

• Verbalized confidence is framing-sensitive and prone to sycophancy, while more elaborate verbalization methods remain subjective [11] [18] [19].Self-consistency sampling estimates confidence from majority-class frequencies across generations, but its multiple-response query cost is prohibitive at scale [17] [10].Source: §2 · p. 2

Exact Extraction via Logit Bias

• Iterative Logit Extraction recovers continuous logits through repeated logit_bias manipulation, but its binary search requires many queries [1].This work instead maps discrete ECE bounds to predefined logit biases, requiring one API call per sample.Source: §2 · p. 2

3 Background: Traditional ECE Computation

• ECE [6] approximates TCE = E[|E[Y|f(X)] − f(X)|] by grouping exact positive-class probabilities into M bins.Here, f(X) is the model’s continuous predicted probability for the positive class.Source: §3 · p. 2
• Despite smooth alternatives such as Kernel Density Estimation [14], calibration metrics require continuous f(X_i) for every sample, which APIs conceal.Source: §3 · p. 2
• The evaluation simulates APIs’ restricted environment, extracting hidden probabilities to compute exact ECE_bin as the oracle for benchmarking.Source: §3 · p. 2

Single-Query Threshold Evaluation

• At temperature 0, a threshold shift b = −ln(t_i/(1−t_i)) and a large shared target-token bias C make the positive-token output indicate f(X_i) ≥ t_i in one query.The shared bias suppresses other tokens and cancels from the logit comparison; the mechanism also holds across semantically equivalent token variants.Source: §4 · p. 3

Partition Scheme and Estimator

• Because one query cannot test both bin boundaries for one sample, ECEblind randomly assigns samples to disjoint subsets and uses adjacent subsets’ threshold indicators to estimate and aggregate bin gaps.It substitutes each bin midpoint for the unknown continuous probability when estimating the local empirical gap.Source: §4 · p. 3

Theoretical Guarantees

• ECEblind’s MSE vanishes as N→∞ when the calibration function is L-Lipschitz and the bin count scales as M ∝ N^α for 0 < α < 1/3.The consistency proof decomposes MSE into a variance component and three bias terms, all of which vanish under this scaling.Source: §4 · p. 3

Optimal Bin Scaling

• The theoretical MSE upper bound is minimized when the bin count scales as M ∝ N^1/5.The derivation is provided in Appendix C.Source: §4 · p. 3

Experimental Setup

• Evaluation uses BoolQ and binarized MMLU, with continuous probabilities from four open-weight models providing white-box ground truth.The models are Qwen-2.5-7B-Instruct, Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3, and Gemma-2-9B-IT; each four-way MMLU question becomes four independent Yes/No questions.Source: §5 · p. 4

Baselines

• Comparators are Verbalized Confidence (K = 1), Monte Carlo Sampling (K ∈{1..8}, T = 1.0), and Iterative Logit Extraction (K ∈{1..8}).All logit-based baselines apply the same C = 50 bias to prevent vocabulary bleeding and ensure a fair comparison.Source: §5 · p. 4

Cost-Error Pareto Frontier

• With one query, ECEblind reduces average MAE from >0.07 for Verbalized Confidence to near 0.01, while sampling plateaus at 0.02 with K = 8.Iterative logit extraction slightly surpasses ECEblind at K ≥6 but requires at least 6x the query cost; the prose reports that ECEblind matches its performance at K = 5.Source: §5 · p. 4, Figure 1 · p. 4

Diagnostic Interpretability

• The Blind Calibration Curve plots signed local gaps by threshold interval, with positive values indicating overconfidence and negative values indicating underconfidence.For Llama-3.1-8B on MMLU, it shows underconfidence at lower probabilities and overconfidence in higher regions; the absolute sum of its bars equals the final ECEblind score.Source: §5 · p. 4, Figure 2 · p. 4

6 Conclusion

• ECEblind uses logit_bias to evaluate exact probability thresholds with strictly one query per sample.This approach addresses the challenge of evaluating calibration when API providers restrict access to continuous probabilities.Source: §6 · p. 4
• ECEblind matches the accuracy of expensive baselines at a fraction of the cost.The conclusion reports this empirical result alongside the estimator’s asymptotic consistency with True Calibration Error.Source: §6 · p. 5
• The method avoids the prohibitive costs of multi-query extraction or sampling and the inaccuracies of verbalizing methods.Its estimator is proved asymptotically consistent with True Calibration Error.Source: §6 · p. 5
• ECEblind offers the research community a mathematically rigorous, economical framework for independent auditing of commercial foundation models.Source: §6 · p. 5

A Limitations and Future Work

• The method depends on APIs exposing logit_bias and currently addresses binary, not multi-class, calibration.Providers have restricted or removed the parameter, and mapping multi-class calibration metrics to single API queries remains open.Source: A Limitations and Future Work · p. 10
• The one-query-per-sample estimator does not determine how to allocate larger query budgets, and next-token probabilities may not reflect factual confidence.The latter limitation concerns whether probability mass on a target token such as “True” captures confidence in the underlying claim [9] [10].Source: A Limitations and Future Work · p. 10
• Clamping with C = 50 suppresses probability mass on alternative valid synonyms, while simulated API conditions may omit production behaviors.The clamping effect also applies to the white-box oracle; live APIs may add prompt formatting, safety filters, or dynamic routing.Source: A Limitations and Future Work · p. 10
• The empirical oracle remains subject to finite-sample and binning biases, and the framework’s M ∝ N^1/5 bin scaling limits resolution on small datasets.For BoolQ’s 3,270-example validation split, the stated optimal bin count is M = 5.Source: A Limitations and Future Work · p. 10

B Detailed Proofs for Estimator Bounds and Consistency

• The proof defines the estimator over equal-width bins and randomly assigns the dataset to disjoint subsets of approximately equal size.The probability space [0, 1] is partitioned into M bins, and each subset contains floor(N/M) or floor(N/M)+1 samples.Source: B Detailed Proofs for Estimator Bounds… · p. 11

B.1 Expected Value and Variance of the Local Empirical Gap

• A local empirical gap has expectation equal to the bin’s trapezoidal target, and TCE decomposes across the disjoint bins.The proof also uses independence of adjacent subsets when bounding the variance of a gap.Source: §B.1 · p. 11

B.2 The 3-Term Bias Decomposition

• The estimator’s expected deviation from TCE decomposes into statistical, trapezoidal, and binning biases.The trapezoidal term reflects replacing predictions with bin midpoints; the binning term reflects discretizing the probability space.Source: §B.2 · p. 12

B.3 Variance Bound via Efron-Stein Inequality

• The estimator’s variance on N samples split into M disjoint subsets is strictly bounded by 8M^2/N.The proof uses the Efron-Stein inequality to bound the effect of replacing one sample.Source: §B.3 · p. 13

Final Proof of Theorem 1 (Estimator Consistency)

• The proof combines the squared-bias and variance bounds using the MSE decomposition, with the variance absorbed into the squared-bias bound.The total absolute bias is bounded by summing the three individual bias bounds before squaring.Source: §B.4 · pp. 13–14
• With M ∝ N^α chosen so both MSE-bound exponents are strictly negative, the MSE vanishes as N → ∞.The supplied passage does not preserve the exact range of α.Source: §B.4 · p. 14
• The theorem assumes an L-Lipschitz true calibration function, which need not follow from a Lipschitz model probability mapping because calibration depends on the dataset’s conditional density [4] [14].The passage describes this Lipschitz premise as standard in theoretical calibration literature for deriving finite-sample bounds.Source: §B.4 · p. 14

C Derivation of the Optimal Bin Scaling

• The derivation differentiates the MSE upper bound with respect to M, sets the derivative to zero, and solves for the optimal bin count.Source: C Derivation of the Optimal Bin Scaling · p. 14
• The ECEblind estimator’s MSE upper bound is asymptotically minimized by scaling the number of bins as N^1/5, balancing subset noise against discretization bias.Source: C Derivation of the Optimal Bin Scaling · p. 14
• Because the upper bound’s tightness is unknown, M ∝ N^1/5 may be suboptimal in practice, but any scaling exponent 0 < α < 1/3 still makes MSE vanish as N → ∞.Source: C Derivation of the Optimal Bin Scaling · p. 15

D.1 Standard ECE Formulation and the Optimal Number of Bins

• Standard empirical binned ECE partitions predicted positive-class probabilities in [0, 1] into M disjoint bins and counts the samples in each bin.Source: §D.1 · p. 16
• Within each bin, acc(Bm) is the average model confidence.Source: §D.1 · p. 16
• A small bin count obscures local calibration errors, while a large count leaves bins sparse; information-theoretic analyses [4] formalize this bias-variance trade-off.Source: §D.1 · p. 16
• For the ECEbin oracle, the experiments use M ∝ N^1/3 as derived by [4], giving M = 15 for BoolQ (N = 3,270) and M = 38 for MMLU (N = 56,168).Source: §D.1 · p. 16

The Restricted Target Space

• The restricted target space addresses vocabulary bleeding by applying C = 50 to token variations for both positive and negative answers, suppressing irrelevant vocabulary.The valid-token sets account for tokenizer variants such as “True,” “ true,” and “TRUE.”Source: §D.2 · p. 16

The Oracle (White-Box Ground Truth)

• The Oracle baseline computes continuous confidence from native hidden logits using a restricted softmax over valid positive and negative tokens.That exact probability is passed to the standard ECEbin equation.Source: §D.2 · p. 16

The Restricted Target Space

• At T = 0, ECEblind outputs a positive token if and only if the model’s original confidence p is at least threshold t, enabling the indicator test in one query.The method shifts positive-token logits by b = −ln(t/(1 − t)); the constant bias C = 50 suppresses tokens outside the valid answer sets.Source: §D.2 · pp. 16–17

Verbalized Confidence (1 Query)

• Verbalized Confidence uses one query to elicit a 50–100% confidence statement, parses the probability, and cannot apply the C = 50 target bias.Source: §D.2 · p. 17

Sampling / Self-Consistency (K Queries)

• Sampling estimates positive-answer probability from K generations at T = 1.0, using C = 50 to restrict the target space and isolate sampling variance.Source: §D.2 · p. 17

Iterative Logit Extraction [1] (K Queries)

• Iterative Logit Extraction [1] uses K greedy queries to binary-search shifts within [−15, +15] for the output-flip boundary, then estimates confidence as σ(−b∗).Source: §D.2 · p. 17

E Extended Results and Breakdown

• Under a 1-query budget, ECEblind nearly matches the Oracle on Llama-3.1-8B BoolQ (0.085 vs 0.084), Mistral-7B MMLU (0.340 vs 0.339), and Gemma-2-9B MMLU (0.314 vs 0.313).Table 1 reports ECE estimates for each method across all evaluated models and datasets.Source: E Extended Results and Breakdown · p. 17, Table 1 · p. 18
• Verbalized confidence is erratic and can overestimate Llama’s MMLU error by nearly 3×, while self-consistency still overestimates calibration error and is less accurate than ECEblind on almost every pair at K = 8.The reported ECE values use five uniform bins and are rounded to three decimal places.Source: E Extended Results and Breakdown · p. 17, Table 1 · p. 18
• Carlini et al.’s binary search is highly inaccurate at K ≤ 3, typically needs 4 to 5 queries to match ECEblind’s single-query precision, and converges to the Oracle at K = 8.Source: E Extended Results and Breakdown · p. 18

F Extended Blind Calibration Curve and Interpretation

• Standard reliability diagrams can mislead because a large confidence–accuracy gap affects ECE according to the sample mass in that bin, which must be manually reweighted.In the K = 1 black-box setting, exact sample probabilities and their confidence-bin masses cannot be extracted; ECEblind instead sweeps a decision boundary and records the local empirical gap.Source: F Extended Blind Calibration Curve and… · p. 19
• The Blind Calibration Curve’s bars show each region’s density-weighted contribution to net error, and the absolute sum of the bars equals the final ECEblind score.Source: F Extended Blind Calibration Curve and… · p. 19
• Positive curve values indicate overconfidence and negative values indicate underconfidence; Figures 3 and 4 compare these curves with white-box reliability diagrams for BoolQ and MMLU.The comparison illustrates that visually large gaps in sparse bins can correspond to negligible penalties in the density-weighted curve.Source: F Extended Blind Calibration Curve and… · p. 19, Figure 3 · p. 20, Figure 4 · p. 21