PaperlayerView exact revision on arXiv

Source-linked AI summary

Epistemic Learning from Imprecise Annotation

Kaizheng Wang, Siu Lun Chau

arXiv:2609.34285v1cs.LG

TL;DR

Imprecise annotations can leave several label distributions plausible, while common learning methods often reduce them to one prediction and do not carry unresolved alternatives forward. The paper learns predictive sets from credal supervision using POCC and relaxed credal labels. Across its evaluated supervision settings, POCC achieves a favourable balance of accuracy, calibration, and selective classification.

  • Problem

    Imprecise annotations can support several plausible targets, but single-target supervision does not explicitly preserve which alternative label distributions remain plausible.

    Source: §1 · pp. 1–2

  • Method

    The paper learns sets of predictive distributions from credal labels and instantiates this approach with POCC's pessimistic and optimistic heads.

    Source: §1 · p. 2

  • Results

    Across annotator disagreement, teacher predictions, and label smoothing as a controlled proxy, POCC achieves a favourable overall balance of accuracy, calibration, and selective classification.

    Source: §1 · p. 2, §6 · p. 14

  • Takeaways & Limitations

    The predictive credal set retains alternatives left unresolved by supervision, with its spread serving as an uncertainty score.

    Source: §1 · p. 2

    The guarantee assumes almost-sure conditional coverage, which need not hold for credal labels from finite annotations or imperfect teacher predictions.

    Source: §6 · p. 14

Abstract

from arXiv · show

Imprecise annotations may support several plausible labelling distributions, yet learning methods often resolve this ambiguity into a single predictive distribution. This can obscure what the annotation evidence leaves unresolved. We introduce epistemic learning from credal supervision, a framework that uses convex sets of plausible labelling distributions, called credal sets, as supervision and learns sets of predictive distributions. We instantiate the framework with the pessimistic--optimistic credal classifier (POCC), which combines a shared backbone with two classification heads trained to minimise worst-case and best-case losses over the supervision sets. Their outputs define a predictive credal set whose spread provides an uncertainty score. We also show how credal labels can be obtained through a simple relaxation of existing probabilistic labels, reducing commitment to their precise probability assignments. This construction admits closed-form inner optimisation under cross-entropy loss, enabling efficient training. Assuming the supervision sets contain the true conditional label distributions, and other regularity assumptions, we establish a finite-sample generalisation bound for the averaged predictor with an explicit penalty for supervision imprecision. We evaluate POCC using human annotator disagreement and teacher predictions, alongside label smoothing as a controlled proxy for annotation imprecision. Across these settings, POCC achieves a favourable balance of predictive accuracy, calibration, and uncertainty-based selective classification versus competitive baselines.

1 Introduction

Imprecise annotations can support multiple plausible targets, but single-target supervision may erase distinctions in their support and leave unresolved alternatives out of predictions. Epistemic learning from credal supervision carries those alternatives into predictive sets, with POCC and relaxed credal labels as practical components.

Motivation

  • A single target can conflate unanimous agreement with substantial disagreement, while soft labels still do not specify which alternative distributions remain plausible.The introduction frames this as a reason to represent what the annotation evidence supports and leaves unresolved.Source: §1 · pp. 1–2

Proposed method

  • POCC uses a shared backbone and pessimistic and optimistic heads to learn worst-case and best-case objectives, whose outputs define a predictive credal set.The spread between head outputs supplies an uncertainty score.Source: §1 · p. 2, Figure 1 · p. 3

Framework

  • The framework learns sets of predictive distributions from sets of admissible label distributions, retaining unresolved annotation evidence at prediction time.This is the stated objective of epistemic learning from credal supervision.Source: §1 · p. 2

Learning-theoretic analysis

  • Under conditional coverage and boundedness assumptions, the averaged predictor has a finite-sample generalisation bound with an explicit penalty for supervision imprecision.POCC's inner optimisation has closed-form solutions under cross-entropy loss for the proposed credal-label construction.Source: §1 · p. 2

Empirical evaluation

  • Across annotator disagreement, teacher predictions, and label smoothing as a controlled proxy, POCC achieves a favourable balance of accuracy, calibration, and uncertainty-based selective classification versus competitive baselines.The evaluation uses human annotator disagreement and teacher predictions to construct credal labels.Source: §1 · p. 2

2 Background

The background distinguishes ordinary probabilistic classification from learning with imprecise annotations and from richer representations of predictive uncertainty. Its central contrast is between a single target or prediction and sets that preserve unresolved alternatives.

Classical Supervised Classification

  • Standard supervised classification learns a probabilistic classifier by minimising population risk, with finite-data training minimising empirical loss and generalisation analysis studying their gap.The background gives cross-entropy as an example loss.Source: §2 · p. 3

Learning from Imprecise Annotations

  • Imprecise annotations may leave several conditional label distributions plausible, unlike label-noise formulations or multi-label learning; missing-label recovery is outside the paper's scope.A missing annotation supplies no direct label constraint.Source: §2 · p. 3
  • Soft labels and label smoothing provide a single probability vector, whereas credal labels retain a set of admissible distributions and leave some target probabilities unresolved.Credal labels have also been used as pseudo-labels in semi-supervised and self-supervised learning.Source: §2 · p. 4

Representing Predictive Epistemic Uncertainty

  • Second-order representations describe uncertainty about which probability vector is warranted, including distributions over vectors or sets of plausible vectors.A credal set specifies admissible distributions without assigning probabilities to its members.Source: §2 · p. 4

Related work

  • Bayesian, ensemble, evidential, and credal predictors represent predictive uncertainty in different ways, but these representations alone do not specify how imprecise annotations should constrain it.The background notes that much existing credal learning still returns ordinary probabilistic predictors.Source: §2 · p. 4

3 Epistemic Learning from Credal Supervision

The section formalises imprecise supervision as sets of conditional label distributions, then uses the extrema of the resulting risk range to train a predictor set. It also relaxes common probabilistic targets into credal labels and derives efficient cross-entropy optimisation.

3 Epistemic Learning from Credal Supervision

  • A credal Markov kernel assigns each input a non-empty, closed, convex set of compatible conditional label distributions; singleton sets recover hard and soft labels.With fixed input marginal, these constraints induce a non-empty, convex, weakly compact joint credal set and a compact interval of population risks whose endpoints are attained.Source: §3 · pp. 4–5

3.1 Pessimistic–Optimistic Credal Classifier

  • POCC avoids training a separate predictor for each admissible law by using the two endpoints of the credal risk profile as complementary objectives.The family of admissible laws is generally uncountable, so sampling tasks can add computational overhead.Source: §3.1 · p. 5
  • The pessimistic and optimistic predictors minimise worst-case and best-case risks over admissible joint distributions, retaining both views rather than selecting a single compromise.This represents consequences of unresolved annotation evidence.Source: §3.1 · p. 5
  • A shared backbone feeds two classification heads, and their predictions define the line segment of mixtures in the predictive credal set.When a single distribution is needed, the method uses the midpoint; the line segment collapses to a singleton when the heads agree.Source: §3.1 · p. 6
  • POCC uses the predictive-set spread as an epistemic uncertainty score, with maximum mean imprecision under total variation reducing to the distance between head outputs.The score is available directly from the two heads without further optimisation.Source: §3.1 · p. 6

3.2 Practical Credal-Label Construction and Efficient Optimisation

  • Credal labels relax a reference probability vector by retaining the most-supported class probability as a lower bound and leaving remaining mass unresolved.The relaxed set contains the original target, while permitting alternatives to its precise probability assignments.Source: §3.2 · p. 7
  • The relaxation is an ϵ-contamination set: every admissible distribution reserves α mass for the most-supported class and leaves the remainder free to reallocate.For fixed most-supported class, decreasing α enlarges the set; a one-hot reference target yields a singleton.Source: §3.2 · p. 7
  • Under cross-entropy, each head's extremal target assigns the unrestricted mass to one class, so the inner problems require an O(K) scan without sampling or iterative optimisation.The additional model cost primarily comes from POCC's second head.Source: §3.2 · pp. 7–8

4 Learning Guarantees under Credal Supervision

The guarantees relate POCC’s averaged predictor to true risk when the credal supervision contains the true conditional label distribution. The bound makes the cost of unresolved supervision explicit under stated regularity conditions.

Coverage of the True Conditional Distribution

  • Coverage requires the true conditional distribution to belong to Q(x) almost surely; it is distinct from, and does not follow merely from, i.i.d. sampling.Coverage relaxes exact distributional supervision to set membership and isolates annotation-constraint validity from misspecification.Source: §4 · p. 8

Generalisation of the Averaged Predictor

  • With probability at least 1 − δ, Theorem 5 bounds the averaged predictor under coverage, a fixed credal kernel, i.i.d. inputs, non-empty F∆, and bounded cross-entropy and head-loss range.The theorem also assumes a common measurable set on which the stated bounds hold.Source: §4 · p. 9
  • The bound separates model complexity, finite-sample variation, optimisation error, and supervision imprecision, whose penalty shrinks with more precise credal labels and vanishes for singleton sets.Different admissible conditional distributions can assign different risks to the same predictor, even when coverage holds.Source: §4 · p. 9

Specialisation to our Credal Construction

  • For Qα,j(x), coverage requires p0,j(x) ≥ α(x), and bounded logits yield an imprecision penalty of 2A E[1 − α(X)].Increasing α narrows the credal label and reduces the penalty, but strengthens the coverage requirement.Source: §4 · p. 9
  • The guarantee concerns cross-entropy risk of the averaged predictor; coverage or calibration of the predictive credal set requires separate analysis.Containing the reference target qL(x) alone does not establish coverage of p0(x).Source: §4 · p. 9

5 Experiments

Across the reported supervision settings, POCC is evaluated for accuracy, calibration, and uncertainty-based selective classification against adapted uncertainty baselines. It achieves the highest BQS on the three annotator-disagreement datasets and a favourable balance across settings, while teacher-prediction results also favor POCC across architectures.

Evaluation and Baselines

  • Evaluation uses annotator disagreement, teacher predictions, and label smoothing as a controlled proxy, measuring ACC, ECE, and selective-classification AUARC over 10 seeds.Higher ACC and AUARC and lower ECE are better; BQS averages their separately min-max-normalised scores.Source: §5 · p. 10
  • Baselines use the same underlying annotation information as POCC, with probabilistic targets for adapted methods and credal labels for POCC; annotator labels use their empirical class distribution.The teacher-prediction setting uses a teacher’s soft predictions as distributional labels.Source: §5 · pp. 10–11

Setup

  • For annotator-disagreement experiments, CIFAR-10H provides about 50 annotations per image; MiceBone and TreeVersity use ImageNet-pretrained DenseNet-121 models.CIFAR-10H evaluation uses the last 10,000 samples of the original CIFAR-10 training set to avoid data leakage.Source: §5 · p. 10

Annotator Disagreement

  • POCC achieves the highest BQS across all three annotator-disagreement datasets; MCDO and Decali have lower ECE but substantially worse ACC and AUARC.Figure 2 compares BQS under annotator disagreement across distinct datasets; Table 1 reports CIFAR-10H results.Source: Figure 2 · p. 11, Table 1 · p. 11, §5 · p. 11

Teacher Predictions

  • With T = 2.5, POCC achieves the best mean ACC, ECE, and BQS across three teacher architectures and both datasets; at T = 1, it leads mean BQS.Figure 3 compares BQS across ResNet-20, ShuffleNetV2-0.5×, and MobileNetV2-x0-5; Table 2 covers ResNet-20.Source: §5 · p. 11, Figure 3 · p. 12, Table 2 · p. 12

Discussion

  • Across the three supervision settings, POCC has the highest mean BQS in most evaluated configurations, though baselines sometimes perform better on individual metrics.Additional analyses report statistical ranking first in balanced performance and robustness to the choice of EU measure.Source: §5 · p. 13

6 Conclusion and Future Work

The conclusion presents epistemic learning from credal supervision as a way to retain unresolved annotation information in predictions. POCC combines complementary heads with closed-form inner optimisation; its averaged predictor has a finite-sample risk bound, and evaluations show a favourable overall balance across accuracy, calibration, and selective classification.

6 Conclusion and Future Work

  • Across annotator disagreement, teacher predictions, and label smoothing as a controlled proxy, POCC achieves a favourable overall balance of accuracy, calibration, and selective classification.The conclusion reports this result across the evaluated supervision settings.Source: §6 · p. 14
  • The analysis assumes almost-sure conditional coverage, which need not hold for credal labels built from finite annotations or imperfect teacher predictions.The conclusion proposes probabilistic coverage or compatibility in conditional expectation as directions for future analysis.Source: §6 · p. 14

A Related Work

  • Existing deep-learning credal predictors assume precise annotations, whereas this work develops credal supervision for unresolved annotation evidence.The conclusion identifies POCC as one instantiation of the framework.Source: §6 · p. 13, A Related Work · p. 23

B.1.1 Setup and Standing Assumptions

  • The induced family of joint laws is a non-empty, convex, weakly compact credal set whose admissible risks form a compact interval.The result assumes a Polish input space and an integrability condition on the predictor’s per-class losses.Source: §B.1.1 · p. 24

B.1.3 Formal Proof of Proposition 2

  • Convexity of the induced joint credal set makes every risk between its attained best- and worst-case endpoints admissible.The proof obtains intermediate risks by mixing endpoint-attaining joint laws.Source: §B.1.1 · p. 24, §B.1.3 · p. 26

B.1.2 Reparametrisation by Selections

  • The affine reparameterisation maps joint laws with input marginal PX to conditional label distributions, with admissible conditionals given by selections of Q.The mapping is a bijection onto the simplex-valued measurable functions, up to PX-almost-everywhere equality.Source: §B.1.2 · p. 24

B.2 Proof of Proposition 3

  • For the relaxed credal labels, cross-entropy optimisation reduces to allocating unrestricted target mass to classes with extremal predicted probabilities.The largest loss corresponds to a least-probable class; the smallest corresponds to a most-probable class.Source: §B.2 · p. 26

B.3.1 Setup and Standing Assumptions

  • Under the stated assumptions, POCC’s generalisation bound includes explicit terms for annotation imprecision and finite-sample estimation error.The bound concerns the averaged predictor and relies on the assumptions referenced in the theorem statement.Source: §B.3.1 · p. 28

B.3.2 Elementary Ingredients for the Proof

Together, these parts present proof ingredients for credal supervision and describe experimental setups using annotator disagreement, teacher-predicted labels, and label smoothing as sources of annotation imprecision.

B.3.2 Elementary Ingredients for the Proof

  • Under conditional coverage, the true risk lies between the optimistic and pessimistic risks, and uniform oscillation bounds both ambiguity gaps by Ld_Q.The section uses a total-variation–oscillation inequality to control how far either endpoint can lie from the true risk.Source: §B.3.2 · pp. 28–30
  • Under conditional coverage, Proposition 10 establishes a population-risk bound for the averaged two-head predictor for every parameter choice.This provides the bridge from the joint credal objective to the true risk of the averaged prediction.Source: §B.3.2 · p. 30
  • With probability at least 1 − δ, Theorem 11 gives uniform convergence of the joint objective under i.i.d. sampling and uniform boundedness.The result does not require a finite or countable parameter class, provided the relevant suprema are measurable and the Rademacher complexity is well defined.Source: §B.3.2 · p. 31

C.3.1 Annotation Imprecision: Annotator Disagreement

  • Annotator-disagreement experiments use CIFAR-10H, MiceBone, and TreeVersity, with ResNet-18 on CIFAR-10H and ImageNet-initialised DenseNet-121 on the other datasets.CIFAR-10H is evaluated on the last 10,000 CIFAR-10 training images, which do not overlap the CIFAR-10H data; MiceBone and TreeVersity use an 80%:20% split. MCDO and LbBnn are excluded on those two datasets because of incompatible pretrained models and out-of-memory errors, respectively.Source: §C.3.1 · p. 38

C.3.2 Annotation Imprecision: Teacher Predictions

  • Teacher-predicted soft labels are formed by applying temperature-scaled softmax to teacher logits, with T ∈ {1.0, 2.5} and three teacher backbones tested on CIFAR-10 and CIFAR-100.Experiments use an 80%:20% train–test split; CIFAR-10 uses ResNet-18, while CIFAR-100 uses WRN-28-4 with five-epoch linear warmup followed by cosine decay to 10^-6.Source: §C.3.2 · p. 39

C.3.3 Annotation Imprecision: Label Smoothing

  • Label smoothing constructs soft labels from hard one-hot labels, with smoothing coefficients 0.05, 0.1, 0.15, and 0.2; credal supervision is then obtained via equation (8).CIFAR-10 uses ResNet-18 and cosine annealing; CIFAR-100 uses WRN-28-4 with five-epoch linear warmup and cosine decay to 10^-6, and both use an 80%:20% train–test split.Source: §C.3.3 · pp. 39–40

C.4 Compute Resources

The appendix lists the computing resources used and describes the metrics for evaluating calibration, selective classification, and balanced performance.

C.4 Compute Resources

  • All experiments use the computing resources specified in Table A3.The supplied passage does not enumerate the resource specifications.Source: §C.4 · p. 40, Table A3 · p. 40

C.5 Evaluation Metrics Implementation

  • Expected calibration error measures the average absolute difference between accuracy and confidence across 10 equally spaced bins.Probabilistic predictions are grouped into bins of equal confidence range.Source: §C.5 · p. 40
  • Selective classification evaluates uncertainty by rejecting high-uncertainty instances and measuring accuracy on retained samples as rejection increases.AUARC summarizes the accuracy–rejection curve; larger values indicate better selective classification performance.Source: §C.5 · pp. 40–41
  • BQS averages min-max-normalised accuracy, 1 − ECE, and AUARC across compared methods for each seed and dataset.The paper reports mean and standard deviation across seeds.Source: §C.5 · pp. 40–41

D Additional Experiment Results

Additional experiments report performance and accuracy–rejection curves across annotator disagreement, teacher predictions, and label smoothing. POCC’s reported results retain a strong balanced-performance pattern across these settings.

Additional experiment results

  • The additional comparisons cover annotator disagreement, teacher-prediction imprecision, and label smoothing, with corresponding results in Tables A4–A6 and curves in Figures A1–A3.The accuracy–rejection curves are shown for annotator disagreement, teacher predictions, and varying label-smoothing ϵ, respectively.Source: D Additional Experiment Results · p. 41, Table A4 · p. 41, Figure A1 · p. 42

Teacher-prediction imprecision

Label-smoothing imprecision

  • Under label smoothing, POCC has the highest mean BQS in seven of eight settings and ranks second for ϵ = 0.1 on CIFAR-10.The exception is attributed to marginal differences in accuracy and AUARC; statistical tests rank POCC first across all settings.Source: Table A6 · p. 44

E Evaluation on Statistical Significance Tests

Pairwise significance tests rank POCC highest in balanced performance across three annotation-imprecision settings. Against baselines trained on collapsed hard labels, POCC also achieves the highest reported balanced scores across both evaluated settings.

E Evaluation on Statistical Significance Tests

  • The pairwise tests use 10 independent runs and mark a method as outperforming another when the one-sided Wilcoxon test is significant at p < 0.05.BQS-ST averages normalised net win–loss scores across accuracy, calibration, and uncertainty quantification.Source: E Evaluation on Statistical… · p. 46
  • POCC achieves the highest BQS-ST scores across annotator disagreement, teacher prediction, and label smoothing, indicating the best reported trade-off across accuracy, calibration, and uncertainty quantification.Figures A4–A6 present the detailed statistical analyses for these scenarios.Source: E Evaluation on Statistical… · p. 46, Figure A4 · p. 47, Figure A5 · p. 49

F Comparison with Baselines Trained on Collapsed Labels

  • For collapsed-label comparisons, baselines train on the majority-vote one-hot label derived from each soft label, with results averaged over 10 runs.The collapsed-label baseline is excluded from knowledge distillation and label smoothing for the stated reasons that collapsing removes teacher soft predictions or recovers the original one-hot label.Source: F Comparison with Baselines Trained on… · p. 51
  • Across both evaluated settings, POCC achieves the highest BQS and BQS-ST among baselines trained on collapsed hard labels.The authors report this as the best trade-off among prediction accuracy, calibration, and uncertainty quantification.Source: F Comparison with Baselines Trained on… · p. 51

G Evaluation with Normalised AUARC

Across annotator-disagreement, teacher-prediction, and label-smoothing settings, POCC achieves the best reported trade-off among accuracy, calibration, and uncertainty quantification. The evaluation uses normalised AUARC to assess selective classification relative to each model’s unrejected accuracy.

G Evaluation with Normalised AUARC

H Evaluation with EU–Prediction Error Correlation

The evaluation examines whether epistemic uncertainty tracks prediction error and compares two uncertainty measures. POCC shows the best reported balance across the tested annotation settings, while maximum mean imprecision performs better and more efficiently than entropy difference on selective classification.

H Evaluation with EU–Prediction Error Correlation

  • Spearman rank correlation measures the association between predicted epistemic uncertainty and cross-entropy loss; ranking avoids dependence on incomparable uncertainty scales.A higher correlation indicates that samples with higher epistemic uncertainty tend to incur greater prediction error.Source: H Evaluation with EU–Prediction Error… · p. 53
  • High uncertainty–loss rank correlation can be misleading when very low accuracy creates varying losses and spurious alignment.The evaluation therefore balances ACC, ECE, and Sp(EU,CE) in BQS and BQS-ST.Source: H Evaluation with EU–Prediction Error… · p. 53
  • POCC consistently achieves the best trade-off among accuracy, calibration, and epistemic-uncertainty quantification across annotator-disagreement, teacher-prediction, and label-smoothing settings.The tables report results for all three scenarios, and the text contrasts POCC with baselines that collapse labels to a single distribution.Source: H Evaluation with EU–Prediction Error… · pp. 53–54, Table A12 · p. 54, Table A13 · p. 55

I.1 Maximum Mean Imprecision

  • Under total variation, maximum mean imprecision reduces exactly to the distance between the two predictive distributions.For the credal set used here, event probabilities are affine in ϵ, so their extrema occur at ϵ ∈ {0, 1}.Source: §I.1 · p. 57

I.2 Shannon Entropy Difference

  • Entropy difference measures the range of Shannon entropy over a credal set, but its maximisation requires numerical optimisation and adds computational overhead.Unlike maximum mean imprecision, the maximisation in equation (A51) has no closed-form solution.Source: §I.2 · p. 57
  • Maximum mean imprecision consistently gives better and more computationally efficient uncertainty quantification than entropy difference on selective-classification AUARC.The comparison covers annotator disagreement, teacher predictions, and label smoothing.Source: §I.2 · p. 57, Table A15 · p. 58, Table A16 · p. 58
  • With entropy difference, POCC has the best BQS across scenarios except CIFAR-10 with label smoothing, where it ranks second behind the 5-member Deep Ensemble.The comparison is shown for annotator disagreement, teacher predictions, and varying label-smoothing ϵ.Source: §I.2 · p. 59, Figure A10 · p. 59, Figure A11 · p. 59

J On the Exclusion of OOD Detection for EU Evaluation

The paper argues that OOD detection is not an appropriate proxy for annotation-induced epistemic uncertainty because the two concern different sources of uncertainty and OOD benchmarks lack a relevant reference signal. It reports OOD results separately, where the POCC ensemble performs best overall.

J On the Exclusion of OOD Detection for EU Evaluation

  • OOD detection concerns whether an input comes from the training input distribution, whereas annotation-induced epistemic uncertainty concerns ambiguity in its conditional label distribution.The two sources are described as conceptually orthogonal: in-distribution inputs can have divergent annotations, while OOD inputs can be annotation-unambiguous.Source: J On the Exclusion of OOD Detection… · p. 60
  • OOD benchmarks provide no reference signal for annotation-induced uncertainty because the test inputs share the training input distribution and uncertainty differences arise from supervision.Thus, OOD performance cannot distinguish genuinely annotation-ambiguous instances from unambiguous ones in this evaluation setting.Source: J On the Exclusion of OOD Detection… · p. 60
  • Selective classification offers an operationally aligned evaluation: abstaining on high-uncertainty predictions should improve accuracy on the retained subset.This tests whether uncertainty ranks annotation ambiguity and prediction errors without claiming that POCC recovers the true credal set.Source: J On the Exclusion of OOD Detection… · p. 60
  • In separate OOD experiments, single-model POCC ranks within the top two, while the POCC ensemble achieves the best overall OOD-detection performance.The experiments use CIFAR-10 as ID data under three supervision settings and compare against several OOD datasets.Source: J On the Exclusion of OOD Detection… · pp. 60–61, Table A18 · p. 61, Table A19 · p. 62
  • The OOD comparison includes single-model methods and multiple-forward-pass approaches, including a five-model POCC ensemble.The tables report AUROC averaged over 10 runs, with methods grouped by model-evaluation approach.Source: Table A18 · p. 61, J On the Exclusion of OOD Detection… · p. 61, Table A19 · p. 62
Loading 2609.34285v1…