PaperlayerView exact revision on arXiv

Source-linked AI summary

Single-Query Black-Box Calibration Auditing via Logit Bias

Roman Plaud, Antoine Saillenfest, Matthieu Labeau, Thomas Bonald, Willem Waegeman

arXiv:2609.05125v1cs.LG

TL;DR

Commercial APIs hide the continuous probabilities needed to calibrate LLM classifiers. The paper uses logit_bias to test exact probability thresholds with one query per sample, yielding a provably consistent binary-task estimator that matches expensive baselines at lower cost. Its applicability depends on APIs continuing to expose logit_bias and on binary classification settings.

  • Problem

    Commercial APIs hide continuous probabilities needed by standard calibration metrics, creating a black-box auditing gap.

  • Method

    The method applies logit_bias threshold shifts to binary target tokens, constructing calibration estimates with exactly one query per sample.

  • Results

    The estimator is asymptotically consistent with TCE and matches expensive baselines at a fraction of the query cost.

  • Takeaways & Limitations

    The approach provides an economical framework for independently auditing commercial foundation models.

  • Takeaways & Limitations

    The method depends on APIs exposing logit_bias and is restricted to binary classification tasks.

Abstract

from arXiv · show

Evaluating the calibration of Large Language Models (LLMs) is critical for their safe deployment as zero-shot classifiers. Yet, commercial API providers increasingly hide the continuous output probabilities required by standard calibration metrics. To bypass this opacity, we demonstrate that any LLM API exposing a logit\_bias parameter can be mathematically manipulated to evaluate exact probability thresholds using strictly one query per sample. Leveraging this mechanism, we introduce a novel and provably consistent estimator of the True Calibration Error for binary tasks. Our approach therefore provides an efficient framework for auditing black-box foundation models.

1 Introduction

Commercial APIs often hide the continuous token probabilities needed for standard calibration metrics, limiting black-box auditing. The paper uses logit_bias to evaluate probability thresholds with one query per sample and introduces a consistent calibration estimator.

  • Standard calibration metrics such as ECE require continuous output probabilities, which commercial APIs frequently restrict.LLM classification commonly relies on probabilities of target tokens such as “True” versus “False” or “A” versus “B”.
  • Logit_bias can be mathematically manipulated to evaluate exact probability thresholds without extracting continuous logits.The approach computes calibration error over a full dataset using threshold queries.
  • The proposed estimator requires exactly one API query per sample and is proved asymptotically consistent with the True Calibration Error.The paper decomposes the estimator’s bias and variance to establish consistency.

2 Related Work

Prior black-box calibration methods either use subjective generative proxies or recover continuous logits through query-intensive extraction. The paper instead targets the discrete thresholds required for binned calibration.

  • The paper positions its single-query estimator as a response to the cost and accuracy limitations of existing black-box proxies.These limitations motivate direct threshold evaluation rather than confidence verbalization or sampling.
  • Verbalization and sampling estimate confidence when exact probabilities are unavailable, but they face framing sensitivity, sycophancy, subjectivity, or high query costs.Reasoning decomposition and multi-prompt aggregation can improve verbalized calibration but remain subjective.
  • Iterative Logit Extraction recovers arbitrary-precision continuous logits through repeated logit_bias queries, making it query-intensive.The proposed work avoids continuous extraction by evaluating predefined discrete calibration thresholds.

3 Background: Traditional ECE Computation

Traditional ECE approximates TCE by binning continuous predicted probabilities, but hidden API probabilities make this computation unavailable. The paper therefore treats exact binned ECE as an oracle for evaluating its estimator.

  • TCE measures the expected absolute difference between conditional accuracy and the model’s continuous predicted probability.For binary data, f(X) is the positive-class probability and Y is the true label.
  • Empirical ECE partitions continuous predictions into M bins and compares each bin’s average accuracy with its average confidence.The bin-level quantities are aggregated to approximate TCE.
  • All calibration metrics considered require knowing each sample’s continuous probability, so hidden API probabilities prevent direct ECE computation.The evaluation simulates this restricted setting and uses exact binned ECE as the oracle benchmark.

4 The Single-Query Black-Box Estimator

The estimator converts logit_bias interventions into exact binary threshold indicators and uses disjoint subsets to approximate binned calibration with one query per sample. Under a Lipschitz calibration function, its MSE vanishes for suitable bin growth, with an upper-bound-optimal scaling of M ∝ N^1/5.

  • Single-Query Threshold Evaluation: The method uses the sigmoid relationship f(X)=σ(z1−z0) to convert probability thresholds into bounds on the difference between positive and negative token logits.The threshold shift is b = −ln(ti/(1−ti)).
  • Single-Query Threshold Evaluation: A targeted logit_bias shift evaluates whether a sample’s hidden confidence exceeds a chosen threshold, producing a binary indicator in one query.A large shared bias suppresses other tokens, while temperature 0 makes the output encode the threshold inequality.
  • Partition Scheme and Estimator: Because one sample cannot test both bin boundaries, the estimator randomly assigns samples to M disjoint subsets and replaces each unknown probability with its bin midpoint.Local bin gaps are estimated from adjacent threshold indicators and aggregated across bins.
  • Theoretical Guarantees: Under an L-Lipschitz calibration function, the estimator’s MSE is bounded and vanishes when M ∝ N^α for 0 < α < 1/3.The proof decomposes error into variance and three bias terms, all vanishing under the stated scaling.
  • Optimal Bin Scaling: The theoretical MSE upper bound is minimized when the number of bins scales as M ∝ N^1/5.This scaling balances statistical subset noise against discretization bias, although the bound’s tightness is not guaranteed.

5 Empirical Evaluation

The evaluation compares ECEblind with proxy and extraction baselines across BoolQ and binarized MMLU using four open-weight models. ECEblind achieves the strongest cost-error trade-off and also provides threshold-level diagnostic interpretation.

  • Experimental Setup: Experiments use BoolQ and binarized MMLU across four open-weight instruction-tuned models, with white-box probabilities providing the calibration ground truth.The models are Qwen-2.5-7B-Instruct, Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3, and Gemma-2-9B-IT.
  • Baselines: Baselines include verbalized confidence, Monte Carlo sampling with K ∈ {1..8}, and iterative logit extraction with K ∈ {1..8}.Logit-based baselines use the same C = 50 bias to reduce vocabulary bleeding.
  • Cost-Error Pareto Frontier: ECEblind reduces average MAE from > 0.07 for verbalized confidence to near 0.01 with a single query per sample.Sampling plateaus at MAE 0.02 even with K = 8 queries per sample.
  • Diagnostic Interpretability: The blind calibration curve plots signed local gaps by threshold interval, with positive values indicating overconfidence and negative values indicating underconfidence.For Llama-3.1-8B on MMLU, lower probabilities are underconfident and higher regions are overconfident; the absolute bar sum equals the final ECEblind score.

6 Conclusion

The paper concludes that ECEblind audits opaque LLM calibration by evaluating exact probability thresholds with one query per sample. It proves consistency with TCE and reports accuracy comparable to expensive baselines at substantially lower query cost.

  • Core Contribution: ECEblind evaluates exact probability thresholds for opaque LLMs using logit_bias and strictly one query per sample.The method targets APIs that restrict access to continuous probabilities.
  • Main Result: The estimator is asymptotically consistent with the True Calibration Error and matches expensive baselines’ accuracy at a fraction of the cost.The conclusion contrasts the method with multi-query extraction, sampling, and verbalization approaches.
  • Implication: ECEblind provides a mathematically rigorous and economical framework for independent auditing of commercial foundation models.This conclusion follows the paper’s theoretical consistency result and empirical cost-efficiency findings.

A Limitations and Future Work

The appendix develops the estimator’s expectation, variance, bias, and consistency bounds through disjoint bin partitions and a three-term error decomposition. Its central limitation is dependence on API support for logit_bias, with the method otherwise scoped to binary classification.

  • Limitations: The methodology depends on commercial APIs exposing logit_bias, which some providers restrict, remove, or ignore.If this feature is fully deprecated, exact black-box calibration auditing via thresholding becomes impossible.
  • Limitations: The technique is restricted to binary classification because a binary probability depends on the difference between two token logits.In multi-class tasks, a token probability depends on all other tokens, making single-bias threshold evaluation nontrivial.
  • Estimator Construction: The dataset is partitioned into M disjoint subsets, and each subset contributes to local empirical gaps associated with adjacent threshold indicators.Disjointness supports the independence arguments used in the variance analysis.
  • TCE Partition: The True Calibration Error is decomposed over M disjoint probability bins, enabling analysis through conditional expectations within each bin.The bins form a complete partition of the probability space [0, 1].
  • Bias Decomposition: The estimator’s expected deviation from TCE is decomposed into statistical, trapezoidal, and binning bias terms.The displayed decomposition is Bias(ECEblind) = Bstat + Btrap − Bbin.
  • Bounds: Under an L-Lipschitz calibration function, binning bias is controlled by the bin width 1/M, while the estimator variance is bounded by 8M^2/N.The variance bound applies to N samples split into M disjoint subsets.

B.4 Final Proof of Theorem 1 (Estimator Consistency)

The proof establishes consistency by showing that the estimator’s variance and bias components vanish under suitable scaling as N grows. This result relies on a Lipschitz assumption for the true calibration function.

  • Proof structure: The MSE decomposes into squared bias and variance, whose bounds are combined to establish Theorem 1.The proof combines the variance bound with three bias terms.
  • Variance control: The variance is bounded by 8M^2/N and is absorbed into the squared bias bound under the stated scaling conditions.The proof identifies the variance order as O(M^2/N).
  • Consistency: The MSE strictly vanishes as N approaches infinity when the number of bins is scaled so the relevant exponents are strictly negative.The proof requires the bin-growth exponent to satisfy the stated asymptotic conditions.
  • Assumption: The consistency argument assumes that the true calibration function is L-Lipschitz, although this property is not guaranteed by the neural network’s Lipschitz continuity.The calibration curve also depends on the conditional density of the dataset, which may create discontinuities.

C Derivation of the Optimal Bin Scaling

The derivation balances discretization bias against statistical variance to select the bin count that minimizes the theoretical estimation-error bound. It yields M proportional to N^1/5, while noting that bound looseness may make this scaling suboptimal empirically.

  • Optimization: The theoretical MSE bound is minimized by differentiating it with respect to the number of bins M.The derivation treats M as the optimization variable in the estimator’s upper bound.
  • Optimal scaling: M ∝ N^1/5 is the resulting asymptotically optimal bin scaling for balancing statistical subset noise against discretization bias.The same scaling is stated as the optimal bin count from the theoretical bound.
  • Caveat: The optimal scaling is not theoretically guaranteed to be tight, so another exponent may produce lower empirical error even though any 0 < α < 1/3 preserves consistency.The limitation concerns practical optimality, not asymptotic vanishing of the MSE.
  • ECE formulation: The standard binned ECE partitions [0, 1] into M disjoint bins, with each bin’s average model confidence computed from its assigned samples.The bin count creates a bias-variance trade-off: small M obscures local errors, whereas large M yields sparse bins.

D.2 Empirical Implementations and the Restricted Target Space

The implementation restricts outputs to valid positive and negative target tokens, then uses logit bias to test probability thresholds with one query per sample. This avoids vocabulary bleeding while supporting comparisons with verbalization, sampling, and iterative extraction.

  • Prompting: Binary prompts constrain outputs to positive or negative answers across datasets such as BoolQ and MMLU.Examples include True/False and Yes/No response formats.
  • Restricted target space: A large positive bias C = 50 is applied to valid positive and negative token variants to suppress irrelevant vocabulary and prevent vocabulary bleeding.The sets V+ and V− include tokenizer variants such as different capitalization or spacing forms.
  • Estimators: The oracle computes exact restricted-softmax probabilities from native logits, whereas the blind estimator evaluates thresholds using a shifted decision boundary.For a threshold tm, the blind method uses bm = −ln(tm/(1−tm)).
  • Single-query evaluation: A threshold shift applied to positive tokens changes the greedy output exactly when the original probability reaches the target threshold.At T = 0, observing a positive token evaluates 1(f(Xi) ≥ tm) in one query.
  • Baselines: Verbalized confidence uses one generation but cannot apply the target bias, while sampling and iterative extraction require K queries per sample.Sampling estimates probability from positive-token frequency; iterative extraction uses binary search over the shift.

E Extended Results and Breakdown

The extended evaluation compares calibration estimators across models and datasets, showing strong accuracy for the one-query estimator and interpretable density-weighted, signed calibration curves.

  • Estimator accuracy: 0.085 vs 0.084 on BoolQ shows ECEblind nearly matching the Oracle for Llama-3.1-8B, despite the one-query budget.The same near-match is also reported for Llama-3.1-8B on MMLU.
  • Estimator accuracy: 0.340 vs 0.339 for Mistral-7B and 0.314 vs 0.313 for Gemma-2-9B show similarly close ECEblind–Oracle estimates on MMLU.These are reported as outstanding configurations for the one-query estimator.
  • Baseline comparisons: At K = 8, self-consistency remains less accurate than the one-query estimator on almost every model-dataset pair, while systematically overestimating calibration error.The comparison highlights both accuracy and query-efficiency differences.
  • Baseline comparisons: Carlini et al.'s binary search is highly inaccurate at K ≤3, typically needs 4 to 5 queries to match ECEblind, and converges to the Oracle at K = 8.Low-budget inaccuracy is attributed to an unresolved search space.
  • Evaluation setup: ECE values are rounded to three decimal places, with the closest estimator to the Oracle bolded and the second closest underlined across five uniform bins.The table compares calibration estimates across models and datasets.
  • Blind Calibration Curves: The Blind Calibration Curve sweeps a decision boundary, records local empirical gaps, and represents density-weighted regional penalties whose absolute sum equals ECEblind.Unlike standard reliability diagrams, it also encodes signed error direction: positive values indicate overconfidence and negative values indicate underconfidence.
Loading 2609.05125v1…