PaperlayerView exact revision on arXiv

Source-linked AI summary

Noisy-Label Calibration with Multiple Unreliable Labels

Keisuke Hanada, Futoshi Futami

arXiv:2610.19543v1stat.MLcs.LG

TL;DR

Noisy annotations can bias calibration assessment, while more examples or majority voting do not necessarily remove the bias. The paper corrects annotator channels to evaluate and recalibrate a fixed binary predictor for clean labels; in its human-label comparison, correction reverses the direction of naive recalibration’s ECE changes.

  • Problem

    When clean evaluation labels are unavailable, noisy annotator responses can bias calibration evaluation and can confound predictor miscalibration with annotator response tendencies.

    Source: §1 · p. 1, §2.2 · p. 4

  • Method

    The paper uses channel-inverted annotations to estimate clean-label ECE and recalibrate a fixed predictor, and estimates unknown channels under conditional-independence and label-orientation assumptions.

    Source: §1 · p. 2

  • Results

    In the fixed-bin human-label comparison, naive recalibration lowered Soft-vote ECE but raised reference-label ECE; correction reversed both changes, while NC-ECE tracked Oracle ECE’s direction.

    Source: §6.3 · p. 12

  • Takeaways & Limitations

    Corrected-loss and histogram fitting provide clean-risk and calibration guarantees, respectively, while using noisy annotations to address the underlying clean-label distribution.

    Source: §7 · p. 12

    The framework assumes score-invariant within-class error rates; identifying unknown channels also requires conditional independence and label orientation, and small channel gaps can amplify correction variance.

    Source: §7 · p. 12

Abstract

from arXiv · show

In medical annotation and crowdsourcing, multiple annotators assign labels to each example, but these labels may be incorrect. Treating such noisy labels as ground truth can bias calibration evaluation. To address this problem, we propose methods for evaluating and recalibrating a fixed binary predictor under the underlying clean-label distribution, using only noisy annotations. First, given annotator-specific noise probabilities, we propose Noise Corrected ECE (NC-ECE), a binned estimator of expected calibration error (ECE) under class-conditional noise. We derive non-asymptotic bounds on the estimator's bias. Then, we propose practical algorithms for estimating unknown noise probabilities under conditional independence and label-orientation assumptions. We also develop loss-based and histogram post-hoc recalibration with clean-risk and calibration guarantees, respectively. Experiments on semi-synthetic benchmarks show that the proposed ECE estimator closely tracks its clean-label oracle counterpart and that the recalibration methods substantially improve latent-label calibration, whereas uncorrected recalibration against noisy annotations can degrade it.

1 Introduction

Noisy annotations can confound predictor miscalibration with annotator response tendencies, so the paper studies evaluating and recalibrating a fixed binary predictor for clean labels using noisy annotations. It develops noise-corrected ECE, methods to estimate unknown annotator channels, and post-hoc recalibration with guarantees.

Motivation

  • Noisy-label evaluation can confound predictor miscalibration with annotator response tendencies, and more examples, averaging, or majority voting need not remove the bias.Recalibration toward annotation agreement can worsen true-label calibration.Source: §1 · p. 1

Calibration evaluation

  • With known annotator-specific noise probabilities, inverting each channel yields corrected residuals whose conditional means support unbiased estimates of clean-label bin residual moments.Summing their absolute values gives the noise-corrected binned ECE estimator.Source: §1 · pp. 1–2

Unknown noise probabilities

  • The paper connects unknown annotator-specific noise-rate estimation from repeated labels to clean-label ECE, accounting for estimation error rather than treating fitted rates as exact.Estimation uses conditional independence and label-orientation assumptions, with score-defined groups for two annotators when needed.Source: §1 · p. 2

Calibration evaluation

  • The ECE analysis separates binning, sampling, and channel-estimation errors; the estimator also supports fixed-bin sample reuse and annotator weights chosen to reduce residual variance.The bounds include sample reuse with fixed bins.Source: §1 · p. 2

Recalibration

  • Corrected-loss and projected histogram fitting provide clean-risk and calibration guarantees, respectively, for post-hoc recalibration.Both evaluation and recalibration target the underlying clean-label distribution using noisy annotations alone.Source: §1 · p. 2

Empirical validation and scope

  • Human annotations show that label choice can reverse recalibration assessments.The paper distinguishes its focus on evaluating a fixed predictor from work that jointly trains a noisy-label classifier or models annotator disagreement.Source: §1 · pp. 2–3

2 Preliminaries

The preliminaries define binary calibration and binned ECE, then formalize annotator-specific class-conditional noise. Under this noise model, raw annotations generally bias calibration evaluation relative to clean labels.

2.1 Calibration in binary classification

  • Calibration requires the conditional positive-label frequency given a prediction score to equal that score, while ECE measures departures from this condition.Binning estimates ECE by grouping similar predictions, but sampling error and within-bin residual cancellation affect the estimate.Source: §2.1 · p. 3

2.2 Multi-annotator setting

  • The multi-annotator setting models each annotator’s false-positive and false-negative rates as constant within each true class, excluding score-dependent errors.The annotator response mean given the score is then a_j + δ_j m(Z), rather than the clean conditional mean m(Z).Source: §2.2 · pp. 3–4
  • Replacing clean labels with one annotator’s labels generally biases ECE toward the true label, and larger samples, averaging, or majority voting do not in general remove that bias.The label-side conditional mean becomes a_j + δ_j m(Z), not m(Z).Source: §2.2 · p. 4

3 Calibration Estimation

The section constructs a corrected-label representation of clean calibration residuals and uses it to estimate ECE from noisy annotations. It gives error bounds for independent and reused samples, with error contributions from binning, sampling, and channel estimation.

Corrected labels and residuals

  • For an informative known channel, transforming an annotation as (A_j − a_j)/δ_j recovers the clean label’s conditional expectation given the prediction score.Subtracting the score gives the corrected residual; fixed convex weights combine annotators’ corrected labels and residuals.Source: §3 · p. 4

Noise Corrected ECE

  • NC-ECE estimates clean-label binned ECE by averaging corrected residuals within evaluation bins and summing the absolute bin residuals.Channel estimates, weights, and the evaluation partition are fitted from a channel-estimation sample independent of the evaluation sample.Source: §3 · pp. 4–5

Independent-sample estimation

  • The independent-sample ECE error bound separates bin approximation, evaluation-sample fluctuation, and channel-estimation error.The stated result assumes class-conditional noise and a Lipschitz clean residual function; smoothness controls the binning term.Source: §3 · p. 5

Convergence rates

  • With fixed K, channel gaps bounded away from zero, and channel-estimation error O_p(ε_nch), equal-width bins yield a convergence rate that includes binning, evaluation-sample, and channel-estimation terms.When n_ch ≍ n_ev ≍ n, the rate is O_p(n^-1/3), matching the order of existing clean-label binning-based ECE bounds.Source: §3 · p. 5

Reusing the estimation sample

  • The paper also bounds ECE estimation when channel fitting and evaluation reuse one sample, provided the predictor and bins are fixed independently of that sample.With fixed K, channel gaps bounded away from zero, and channel error O_p(ε_n), the reused-sample result gives the stated convergence rate with both sample sizes replaced by n.Source: §3 · p. 6

4 Channel and Weight Estimation

The section estimates unknown annotator channels from joint responses, using score groups when two annotators are available and response vectors when there are at least three. It also chooses aggregation weights to reduce residual variance.

4.1 Setting and assumptions

  • Conditional independence given the true label and score, with channels shared across score groups, underpins the channel-identification methods.One annotator, or two annotators without score information, are generally insufficient.Source: §4.1 · p. 6

4.2 Two annotators and two score groups

  • Two annotators’ response-pair means and covariances across two score groups yield a closed-form channel and class-proportion inverse when the groups have different class proportions.The class proportions are estimated, and a score split alone does not ensure that they differ.Source: §4.2 · pp. 6–7

4.3 Three or more annotators

  • With at least three informative annotators, response vectors alone identify channels without score-group separation; a third-order moment and pairwise covariances recover prevalence up to complement.A prescribed label orientation resolves the complement; likelihood estimation uses responses rather than predictions, and a finite EM run need not reach a global maximizer.Source: §4.3 · pp. 7–8

4.4 Choosing aggregation weights

  • Under the stated assumptions and nonzero channel gaps, inverse corrected-noise-variance weights minimize residual variance, with estimated variances used for a practical plug-in rule.The optimality result assumes known channels.Source: §4.4 · pp. 8–9

5 Recalibration

The paper fits post-hoc maps using corrected labels, with corrected-loss fitting targeting clean log risk and projected histogram fitting targeting calibration. The guarantees differ: a log-risk guarantee does not ensure that every fitted parametric map reduces ECE.

5.1 Corrected-loss fitting

  • Corrected-loss fitting uses labels corrected from estimated channels to fit a post-processing map, with a guarantee of clean log risk close to the best map in the chosen family.The excess-risk bound includes sampling, channel-estimation, and optimization errors.Source: §5.1 · p. 9, §5 · p. 9
  • The corrected-loss guarantee is not a guarantee that every fitted temperature or Platt map reduces ECE.The fitting procedure minimizes corrected empirical risk over temperature or Platt scaling maps.Source: §5.1 · p. 9

5.2 Fixed-bin recalibration

  • Projected histogram recalibration uses corrected bin means, projected to probabilities, and has a calibration guarantee under the stated assumptions.Its fitting bins need not match the channel-estimation groups or evaluation bins.Source: §5.2 · pp. 9–10

6 Experiments

The experiments test noisy-label ECE estimation and post-hoc recalibration on toy data, simulated-noise tabular data, and real CIFAR-10N annotations. Across these settings, correction tracks clean-label evaluation and can improve recalibration, while uncorrected fitting can move calibration in the wrong direction.

6 Experiments

  • The experiments cover toy ECE estimation and sample reuse, simulated-noise tabular recalibration, and a real-image stress test without guaranteed theoretical assumptions.The tabular study uses four datasets; the image study uses CIFAR-10N human labels.Source: §6 · p. 10, §6.3 · p. 12

6.1 Toy data

  • NC-ECE errors decrease with sample size, and estimated noise rates perform close to supplied rates in the calibrated toy condition.The comparison uses the calibrated case, whose true ECE is zero; the figure reports MAE scaled by 100.Source: §6.1 · p. 10, Figure 1 · p. 11
  • At N = 8000, reuse and cross-fitting have similar mean errors, both lower than splitting across all three toy conditions.Finite-sample error remains even when true ECE is zero; the calibrated case illustrates permitted sample reuse, not a theoretical ranking.Source: Table 1 · p. 10, §6.1 · p. 11

6.2 Recalibration on tabular data

  • On all four tabular datasets, corrected Platt and fixed-bin methods reduce mean ECE versus Base, but corrected temperature scaling worsens ECE on MEPS.The results use original labels from separate test examples for evaluation.Source: Table 2 · p. 11, §6.2 · p. 12

6.3 Real label noise: a stress test

  • On CIFAR-10N, NC-ECE stays close to Oracle ECE and tracks its direction of change; naive recalibration improves Soft-vote ECE but worsens reference-label ECE, while correction reverses both changes.The noise assumptions are not verified in this stress test.Source: §6.3 · p. 12, Table 3 · p. 12

7 Conclusion and Limitations

The paper uses corrected annotations for ECE evaluation and recalibration, with separate error bounds and guarantees for two fitting approaches. Its framework depends on class-conditional score-invariant error rates and, for identifying unknown channels, conditional independence and label orientation.

7 Conclusion and Limitations

  • Corrected annotations support ECE evaluation and recalibration from noisy responses, with bounds separating binning, sampling, and channel-estimation errors.Corrected-loss fitting provides a clean-risk guarantee; histogram fitting provides a calibration guarantee.Source: §7 · p. 12
  • The framework assumes score-invariant error rates within each true class; identifying unknown channels additionally requires conditional independence given the true label and score, plus label orientation.Small channel gaps can amplify correction variance, and weak separation between group class proportions can destabilize two-annotator estimation.Source: §7 · p. 12
  • Extending the framework to multiclass labels and addressing its assumptions are stated directions for future work.Likelihood rates may not hold after finitely many EM steps.Source: §7 · p. 12

Supplementary Material

The supplement contains proofs, additional statistical results, algorithms, and experimental protocols for the paper’s main sections. Its notation appendix specifies conventions and notes that sparse responses require additional assumptions.

Supplementary Material

  • The supplement collects proofs for Sections 3–5, algorithms and experiments, notation, annotation-model discussion, and additional related work.Appendices C–E establish the results of Sections 3–5; Appendices F and G give algorithms and experiments.Source: Supplementary Material · p. 16

A Notation

  • Notation covers random variables and distributions, calibration and recalibration partitions, corrected labels, residuals, and sample definitions.The appendix also specifies measurability and independence conventions, norm notation, and bin-boundary conventions.Source: A Notation · p. 16
  • The information-sufficiency table distinguishes correction from identification; unknown-channel estimation additionally requires label orientation, and rate results impose regularity conditions.Source: Table 4 · p. 17

B.1 Population and empirical binned errors

The appendix relates population binned ECE to full ECE and develops conditions for correcting annotations and identifying unknown channels. It also explains why fixed aggregation may retain bias and represents calibration errors through corrected moments.

B.1 Population and empirical binned errors

  • For a fixed partition, population binned ECE is no greater than full ECE, so fixed-bin estimation can retain binning error even as sampling error vanishes.Cancellation within a bin can make binned ECE zero while full ECE is positive.Source: §B.1 · p. 17

B.2 Conditional distributions and identification

  • Known-channel correction uses the marginal relation and nonzero channel gaps without requiring annotator independence, whereas unknown-channel identification needs stronger assumptions including conditional independence and orientation.Dependence can violate the product likelihood relation while preserving marginal correction; a shared correction may also fail if channels vary with score.Source: §B.2 · p. 17

B.3 Aggregation before correction

  • For fixed annotator count, majority vote and fixed weighted averages need not eliminate conditional-mean mismatch as evaluation examples increase.Prediction-dependent posterior aggregation is not excluded, but fitted posteriors require their own estimation-error analysis.Source: §B.3 · p. 18

C.1 Weighted representations and corrected moments

  • Under class-conditional noise with nonzero channel gaps, corrected labels have conditional expectation equal to the clean label, yielding unbiased empirical corrected moments on an independent evaluation sample.The weighted representation connects binwise calibration moments to the sum of their absolute values.Source: §C.1 · pp. 18–19

C.2 Independent-sample ECE bound

This section studies when calibration error under noisy labels can be estimated reliably, developing finite-sample and asymptotic guarantees for corrected ECE estimation under independent sampling, sample reuse, and cross-fitting, and examining identifiability and stable inversion using annotators across score groups. Its guarantees depend on conditions such as fixed evaluation bins, sufficient information, and separation.

C.2 Independent-sample ECE bound

  • With independent channel-fitting and evaluation samples, Theorem C.1 bounds finite-sample error and conditional bias; a Lipschitz residual adds a bin-width term for the population ECE bound.The theorem’s first two assertions require no smoothness; the population extension uses the residual smoothness condition.Source: §C.2 · pp. 19–20

C.2.1 Proof of Corollary 1

  • For equal-width bins, M ≍ n^1/3 balances binning and sampling terms, but the rates require the stated channel-error rate and do not cover coefficients fitted on the evaluation sample.For a fixed partition, the smoothness-free bound is relative to the auxiliary binned ECE, not the population ECE.Source: §C.2.1 · p. 20

C.3 Same-sample estimation with fixed bins

  • A uniform finite-class argument permits channels and weights to be fitted on the evaluation sample while retaining a bound for a fixed predictor and partition.The population-ECE extension adds L_r h_B under a Lipschitz residual; the guarantee does not cover a data-fitted partition or predictor, and failed-fit coverage is not conditional on successful fits.Source: §C.3 · p. 21

C.4 Cross-fitting with one common evaluation partition

  • Cross-fitting with one common evaluation partition bounds cross-fitted calibration error by aggregating held-out bin moments, with a Lipschitz term for the population extension.The common partition ensures each fold estimates the same bin moments; the result does not establish calibration for a combined map of fold-specific recalibrators.Source: §C.4 · p. 22

C.5 NC-ECE and histogram plug-in estimators

  • With constant weights and channels, averaging annotations within bins and applying affine corrections gives the same estimator as the histogram plug-in computation.Projection, a different fitting sample, different bins, or a different conditional-expectation estimator can change results; moment analysis establishes no uniform ranking.Source: §C.5 · p. 23

C.6 Aggregation weights and residual variance

  • For known channels, optimal constant weights minimize residual variance by weighting annotators inversely to their average conditional variance; if some variances are zero, minimizers use only zero-variance annotators.Equal, Gap, and Squared-gap rules are alternatives, but residual-variance comparisons do not establish finite-sample ECE mean-squared-error optimality.Source: §C.6 · p. 23

C.6.1 Connection to bin-moment and NC-ECE variance

  • IV weights minimize exact total bin-moment variance and, under nonzero population bin moments, leading asymptotic known-channel NC-ECE variance, but not necessarily finite-sample NC-ECE variance.For K = 2 and n = 1, IV weighting lowers residual variance from 1 to 5/8 while NC-ECE variance rises from 0 to 15/64; absolute bin moments can have finite-sample Jensen bias.Source: §C.6.1 · pp. 24–25

D Channel Identification and Estimation

  • The channel-error bounds can be inserted into calibration bounds after addressing identification and then estimation of the channel parameters.The section distinguishes whether parameters are uniquely determined by the available-data distribution from how accurately they can be recovered from finite samples.Source: D Channel Identification and Estimation · p. 25

D.1 Non-identification without sufficient information

  • The same score-annotation distribution can yield calibration errors 1/8 and 5/32 under different positively oriented one-annotator channels and true-label distributions.Thus population true-label calibration error is not generally identifiable from one annotator’s noisy labels.Source: §D.1 · p. 25
  • With two annotators after discarding scores, continuously varying positively oriented channels can produce the same joint annotation probabilities (0.31, 0.19, 0.19, 0.31).The marginal two-annotator model is therefore not generally identified, and longer optimization cannot recover information discarded with the scores.Source: §D.1 · p. 25

D.2 Two annotators in two score groups

  • Two annotators in two score groups uniquely identify their channels and group class proportions under conditional independence, positive orientation, and distinct group prevalences.Scores need not be calibrated, but the group proportions must differ; the exactly identified model provides no excess moments for testing its assumptions.Source: §D.2 · pp. 26–27

D.3 Concentration and stable inversion

  • Under separation and interior assumptions, the closed-form inverse is feasible and its parameter error is bounded when empirical moments are sufficiently close to their population values.With adequate group counts, parameter error is O_p(n_ch^-1/2); the local stability constants are not automatically observable confidence bounds.Source: §D.3 · pp. 27–28

D.4 Constrained GMM and optimization error

This section presents identification and estimation guarantees for noisy-label models, along with guarantees for corrected-loss and histogram recalibration. It also gives a clean log-loss guarantee for parametric fitting and calibration guarantees for fixed-bin recalibration under noisy labels.

D.4 Constrained GMM and optimization error

  • For fixed L, constrained GMM is quantitatively identified under the stated conditions, and a certified Op(n_ch^-1) optimization gap preserves root-n_ch error order; a global minimizer has zero gap.The identification constant can shrink as group differences or gaps decrease, and a stopping tolerance or stationary point alone does not certify the required global objective gap.Source: §D.4 · p. 29

D.5 Marginal identification with three informative annotators

  • Three informative annotators and a known orientation sign identify the class prevalence and all annotator channels up to global label permutation, even if other annotators are uninformative.An uninformative channel can be estimated in the likelihood model but cannot be inverted as a known channel.Source: §D.5 · pp. 30–31

D.6 Regular likelihood estimators and grouped models

  • For correctly specified, identified finite-alphabet models with fixed K and L, likelihood estimates are consistent and achieve root-n_ch error under interior truth, nonsingular Fisher information, and controlled optimization error.The grouped model's stated sufficient identification conditions are either a positively oriented informative pair with distinct group proportions, or three informative annotators with common orientation; EM convergence alone does not establish global optimality or consistency.Source: §D.6 · pp. 31–32

D.7 Model checks and extensions

  • The results do not guarantee validity under arbitrary selective missingness, dependence, or instance-dependent channels; group-specific channels require informative overlap and common orientation in every group.Moment-based model checks also need separate regularity and calibration, especially near boundaries or weak identification.Source: §D.7 · p. 33

E Recalibration: Statements and Proofs

E.1 Corrected binary losses

  • With known nonzero-gap channels, corrected binary loss recovers clean risk in expectation; with estimated channels, risk error on an independent example is bounded under the stated class condition.The estimated-channel comparison does not require nonzero true gaps.Source: §E.1 · p. 33

E.2 Parametric logarithmic-loss fitting

  • Parametric logarithmic-loss fitting uses temperature-scaled or affine-logit calibrators after clipping inputs, with compact parameter sets ensuring bounded outputs.The bounded probabilities are substantive conditions for the corrected logarithmic loss.Source: §E.2 · p. 34
  • The fitting objective is convex on a convex parameter set but can be unbounded below without the stated compactness.Source: §E.2 · p. 34
  • An approximately minimizing parametric fit has clean log loss at most the best family member’s risk plus optimization error and twice a finite-sample/noise deviation, with high probability conditional on channel data.The guarantee assumes a fitted map whose corrected empirical risk is within ϵopt of the family minimum.Source: §E.2 · p. 34
  • For fixed-dimensional compact parametric families, the complexity is O(nrec^-1/2) when the stated parameter, clipping, and boundedness conditions hold.The constants depend on the fixed bounds and bB.Source: §E.2 · p. 34

E.3 Fixed-bin recalibration

  • The fixed-bin calibration theorem gives a high-probability guarantee simultaneously for every possibly sample-dependent fitted map.The guarantee is conditional and holds under Proposition 2’s hypotheses.Source: §E.3 · p. 35
  • For fixed J and K, positive minimum channel margins, and ech=Op(εnch), projected-histogram ECE is Op(nrec^-1/2 + εnch), without smoothness of m.The projected-histogram bound has no empirical projection-residual term.Source: §E.3 · pp. 35–36

E.4 Evaluation of a fitted predictor

The section gives a conditional held-out evaluation guarantee for a fitted predictor using estimated annotator channels. It also specifies when binning is exact and how the split-sample workflow keeps evaluation separate from fitting.

E.4 Evaluation of a fitted predictor

  • Proposition E.1 bounds held-out evaluation error conditional on fitting information when estimated annotator channels have positive gaps and evaluation examples are independent.The guarantee is conditional on the samples determining the fitted map, weights, and test partition.Source: §E.4 · p. 36
  • A partition separating every distinct output of a finite-output recalibration map makes binned ECE exact, but fixed equal-width bins may not do so and sampling error remains.For a general map, the stated Lipschitz condition gives an approximation term Lg hB.Source: §E.4 · pp. 36–37

E.5 Split-sample workflow

  • The workflow makes pre-evaluation choices independent of Dev; a separate channel sample may replace fitting channels only for the evaluation step.The held-out guarantee is conditional on the samples used to determine the fitted map, weights, and test partition.Source: §E.5 · p. 37

F Algorithms and Implementation

The section describes likelihood updates and implementation choices for estimating annotator channels, then details aggregation weights, computational costs, and experiment-reporting conventions.

F.1 Likelihood updates

  • EM updates use posterior class probabilities and maximize expected complete-data log likelihood; the marginal response likelihood is (1−µ)Li0 + µLi1.Posterior probabilities lie in [0, 1], while corrected labels need not.Source: §F.1 · p. 38, §F.1.1 · p. 38

F.1.1 Marginal Dawid–Skene

  • The marginal two-annotator Dawid–Skene fit is used only as a non-identified control, and its updates do not use predictions..Source: §F.1.1 · p. 38

F.1.2 Shared channels with score groups

  • With score groups, the shared-channel likelihood pools channel updates across examples; fitting group-specific channels is an extension not covered automatically by the shared-channel guarantees.Joint score-bin EM uses the shared-channel likelihood.Source: §F.1.2 · p. 38

F.2 Numerical procedure

  • The numerical procedure uses multiple EM starts, imposes orientation by a global label exchange, and applies estimator-specific acceptance checks.Grouped EM selects the largest-likelihood candidate after orientation and gap screening, then checks convergence and the final gap threshold.Source: §F.2 · p. 38

F.3 Computational cost

  • For R starts and Q iterations, shared-channel and marginal EM require O(RQnK) operations; NC-ECE costs O(nK + M) given bin memberships.The passage also gives storage costs for these procedures and other estimators.Source: §F.3 · p. 38

F.4 Feasible aggregation weights

  • The tabular Main rule averages equally weighted corrected labels for the class proportion, while Equal, Gap, and Squared-gap weights scale as 1, estimated gap, and squared estimated gap.The estimated class proportion is clipped to [10^-4, 1 − 10^-4], and estimated variances are floored before inversion.Source: §F.4 · p. 39

G.1 Evaluation and reporting conventions

  • The reporting conventions distinguish population ECE from empirical oracle references and define repetition bias, MAE, and RMSE from repetition errors.Figure error bars show one Monte Carlo standard error of MAE; Tables 1 and 5 report sample SDs over 200 and 60 repetitions, respectively.Source: §G.1 · p. 39

G.2 Synthetic experiments

The synthetic experiments include Gaussian recalibration and toy ECE estimation with noisy annotations. They report evaluation metrics and the scope of Figure 2, but the supplied passages do not give comparative metric values.

G.2 Synthetic experiments

  • The Gaussian recalibration experiment uses two conditionally independent annotators and total sample sizes 1,500, 3,000, 6,000, and 12,000, with independent channel-fitting, recalibration, and test thirds.The predictor and label relationship are specified by the Gaussian model.Source: §G.2 · p. 39
  • At n = 12,000, Table 5 reports Gaussian recalibration metrics as means ± SD over 60 repetitions; evaluation includes L1(m), log loss, and Brier score.The passage describes temperature, Platt, and projected histogram fitting procedures.Source: §G.2 · p. 39, Table 5 · p. 40
  • In the toy experiment, all 200 joint EM fits succeeded in each reported condition, and Figure 2 reports ECE-estimation MAE (×100) for Shifted and Nonlinear cases.The toy study uses five score-defined channel groups and ten fixed equal-width evaluation bins.Source: §G.2 · pp. 39–40, Figure 2 · p. 40

G.3 Semi-synthetic tabular benchmarks

The experiments describe semi-synthetic tabular benchmarks and a CIFAR-10N stress test for channel estimation, ECE evaluation, and recalibration. The discussion situates these methods among related work and states assumptions and scope boundaries.

G.3 Semi-synthetic tabular benchmarks

  • The tabular benchmarks use six predictor families on four datasets, simulated annotator-specific class-conditional noise, and multiple annotator counts and noise regimes.The primary analysis uses Joint score-bin EM and Main plug-in IV weights; fitting and evaluation use separate data divisions, with specified departures from theoretical guarantee conditions.Source: §G.3 · p. 41
  • Channel fits that fail finite-parameter, estimated-gap, or estimator-specific checks are excluded, across 720 predictor–seed–annotator-count–regime cells per dataset.Pooled summaries cover 2880 cells; recalibration means and sample SDs aggregate ten seed-level means.Source: §G.3 · p. 41

G.4 CIFAR-10N stress test

  • CIFAR-10N tests the procedures without guaranteed noise assumptions, using binary animal-versus-vehicle labels and original CIFAR-10 labels as evaluation references.Marginal Dawid–Skene EM estimates channels from responses alone; the benchmark labels are not used for fitting or selection.Source: §G.4 · p. 42

H.1 Related work

  • The related-work discussion distinguishes this paper’s estimation of unknown annotator-specific channels from prior noise-correction and noisy-label calibration settings.The predictor is fixed independently of channel estimation and recalibration; the model presupposes one meaningful binary outcome, with disagreement interpreted as imperfect measurement.Source: §H.1 · pp. 42–43

H.2 Scope

  • The guarantees have distinct targets: corrected-loss fitting concerns clean risk, histogram fitting concerns calibration, and IV optimality concerns known-channel variance under stated conditions.Human-label results do not verify noise assumptions or clinical validity, and shared evaluation samples limit interpretation of pilot-fit SDs.Source: §H.2 · p. 43
Loading 2610.19543v1…