PaperlayerView exact revision on arXiv

Source-linked AI summary

Hard Cases, Bad Labels: Testing Error Exposure and Error Location in Uncertainty Sampling Under Bounded Label Noise

John Myron Uy

arXiv:2608.13601v1cs.LG

TL;DR

Uncertainty sampling may select informative examples that are also harder to label, raising questions about corrupted-label exposure and error location. This paper separates these mechanisms with controlled comparisons across datasets and noise conditions, finding that label-efficiency benefits and apparent robustness are conditional rather than universal.

  • Problem

    Uncertainty sampling may acquire more corrupted labels and place errors in especially harmful difficult regions, but standard comparisons do not distinguish these mechanisms.

  • Method

    A paired benchmark compares margin-based uncertainty sampling with random sampling across three binary datasets, clean labels, synthetic noise, and exposure-matched random classification noise.

  • Results

    Uncertainty sampling improved clean-label balanced-accuracy efficiency on all three datasets, while noise effects varied by dataset and no universal additional location penalty was detected.

  • Takeaways & Limitations

    Uncertainty sampling can be label-efficient, but its apparent robustness depends on dataset structure, annotation budget, noise process, and evaluation metric.

  • Takeaways & Limitations

    The synthetic difficulty-dependent noise process models neither real human disagreement nor a worst-case bounded-noise adversary.

Abstract

from arXiv · show

Active learning can reduce labeling cost by selecting informative examples, but the most uncertain examples may also be the hardest to label correctly. This study tests whether uncertainty sampling fails because it acquires more corrupted labels or because errors concentrated in difficult regions are especially harmful. Margin-based uncertainty sampling is compared with random sampling under clean labels, random classification noise (RCN), and bounded difficulty-dependent noise on three public binary tabular datasets. The design uses 100 paired seeds, nine expected noise rates from 0 to 0.30, annotation budgets from 20 to 120, and logistic regression with regularization re-selected by cross-validation at every budget. An exposure-matched RCN control aligns mean final acquired corruption, while a clean-label extension reaches budget 400. Under clean labels, uncertainty sampling improved normalized balanced-accuracy area under the learning curve by 1.09 to 1.77 percentage points on all datasets. Difficulty-dependent noise reduced this advantage more than RCN at six of eight rates on Breast Cancer Wisconsin, but at no tested rate on Banknote Authentication or MAGIC Gamma Telescope. Exposure-matched analyses found no corrected evidence for a universal additional penalty from structured error location. On clean MAGIC data, uncertainty sampling improved balanced accuracy while reducing average precision and true-positive rate at fixed false-positive rates. Thus, uncertainty sampling was label-efficient, but its apparent robustness depended on dataset, budget, noise structure, and evaluation metric.

1 Introduction

The introduction frames uncertainty sampling as a tradeoff between information and annotation risk, separating acquired-error exposure from the potential harm of where errors occur. The study tests this distinction across datasets, noise processes, budgets, and evaluation metrics using paired, exposure-matched comparisons.

  • Motivation: Margin-based uncertainty sampling selects examples near the decision boundary because their labels may most influence its placement.These potentially informative examples may also be difficult for annotators and more likely to receive incorrect labels.
  • Motivation: The study separates two mechanisms: uncertainty sampling may expose the learner to more corrupted labels, and structured error location may remain harmful after exposure is aligned.The distinction matters because higher exposure can motivate repeated labeling or escalation, whereas residual location effects concern where errors occur.
  • Study design: The benchmark compares uncertainty and random sampling on three binary tabular datasets under constant-probability RCN and bounded difficulty-dependent noise.It uses paired seeds, complete low-budget learning curves, and budget-specific regularization selection, while reporting results separately by dataset.
  • Study design: An exposure-matched, leave-one-seed-out RCN control tests whether performance differences remain after aligning mean final acquired corruption.The control compares the structured process with independent RCN calibrated to match mean final acquired corruption, without perfectly isolating location in every trajectory.
  • Evaluation: On clean MAGIC data, uncertainty sampling improves balanced accuracy but reduces average precision and TPR at every pre-specified FPR limit.This metric reversal is presented as a bounded contribution of the study.

2 Related Work

Prior work established active learning’s labeling-efficiency promise while showing that measured advantages depend on training choices, randomness, datasets, and budgets. This study extends imperfect-annotation research with a narrower difficulty-dependent noise process, exposure matching, paired tabular experiments, and metrics beyond AUROC.

  • Active-learning foundations: Sequential example selection can reduce labeled data needs, but active-learning conclusions vary with training settings, randomness, dataset, and budget.Lewis and Gale established the labeling-efficiency result, while later work organized active-learning methods and showed sensitivity to regularization and training choices.
  • Noise models: RCN flips binary labels independently at a constant probability, whereas Massart noise allows input-dependent probabilities bounded below one half.The present study uses a narrower non-adversarial process: error probability is a fixed monotone function of difficulty relative to an estimated reference boundary.
  • Noisy active learning: The experiment complements recent noisy-annotation studies through standard logistic regression, tabular datasets, explicit exposure matching, and paired comparisons across the complete annotation path.Related work has examined imbalance with label noise and noise-aware low-budget frameworks across multiple image-benchmark noise processes.
  • Evaluation metrics: Because AUROC may obscure practically relevant operating regions, the study reports balanced accuracy, average precision, and TPR at specified FPR limits instead.MAGIC is additionally evaluated at 1% and 2% FPR because its documentation describes accepting a background rate at those operating points.

3 Methods

The study used paired active-learning trials across three public binary datasets, comparing random and margin-based uncertainty sampling under clean, random, and difficulty-dependent label noise. Logistic regression was re-tuned at each budget, with exposure-matched controls and predefined learning-curve and secondary-metric analyses.

  • Study design: Three public UCI binary-classification datasets varied in size, dimensionality, class balance, and domain.
  • Study design: Each of 100 seeds used an 80/20 stratified pool/test split, pool-only standardization, and paired comparisons within dataset and seed.Test labels were excluded from training, selection, cross-validation, and synthetic-noise construction.
  • Learner and acquisition: Logistic regression with L2 regularization was re-selected by three-fold seeded stratified cross-validation using only labels acquired at each budget.The model was retrained from scratch after selecting inverse regularization strength, optimizing balanced accuracy with ties favoring smaller C.
  • Learner and acquisition: Trials began with 20 trusted clean labels, then compared uniform random sampling with five-example smallest-margin uncertainty batches through budget 120.Sampling proceeded in batches of five and produced 21 recorded checkpoints.
  • Noise construction: Noise conditions included independent random classification noise and bounded difficulty-dependent noise calibrated to preserve mean expected error while increasing with reference-model difficulty.Difficulty was defined using a frozen, independently cross-fitted random-forest reference and was unavailable to the learner and evaluation.
  • Hypotheses and extensions: Exposure-matched RCN targets separated greater corrupted-label acquisition from greater harm caused by corrupted labels near difficult regions.A secondary clean-label Breast Cancer Wisconsin experiment extended the budget from 120 to 400.

4 Results

Uncertainty sampling improved clean-label normalized balanced-accuracy AULC on all three datasets, but noise effects were dataset-specific and did not produce general failure through 30% expected noise. Exposure-matched analyses found no universal additional penalty from difficulty-dependent error location, while clean MAGIC results revealed metric tradeoffs.

  • Clean-label performance: 1.77 points on Breast Cancer, 1.45 percentage points on Banknote, and 1.09 points on MAGIC quantified uncertainty sampling’s clean-label normalized balanced-accuracy AULC advantage.H1 was supported on all three datasets, with positive uncertainty-minus-random differences.
  • Noise effects: Six consecutive rates from 0.025 through 0.20 showed significant Breast Cancer degradation under difficulty-dependent noise relative to RCN, with mean effects from −0.84 to −1.26 percentage points.The 0.25 comparison was inconclusive, while the 0.30 bootstrap interval was below zero but its Wilcoxon result did not survive Holm correction.
  • Noise effects: Banknote had no corrected difference at any rate, MAGIC had no corrected differences, and the cross-dataset hypothesis that difficulty-dependent noise generally reduced uncertainty sampling’s advantage more than RCN was unsupported.Banknote had six of eight negative point estimates with intervals crossing zero; MAGIC had only two negative point estimates.
  • Noise effects: No point-estimate AULC curve crossed below zero, and all six dataset-process crossing estimates were right-censored above 0.30, indicating reduced advantage rather than general failure through 30% expected noise.This conclusion concerns the tested domain and does not imply robustness under higher noise rates.
  • Exposure-matched analysis: No comparison supported H4’s predicted negative direction; instead, difficulty-dependent noise exceeded exposure-matched RCN by 1.38 percentage points at rate 0.10 and 1.89 points at rate 0.15 on MAGIC.The exposure-matched control aligned mean final acquired exposure closely, but the opposite MAGIC results reject a universal claim that concentrated errors are more harmful.
  • Secondary metrics: On clean MAGIC at budget 120, uncertainty sampling improved balanced accuracy by 1.12 percentage points but reduced average precision by 2.21 points and TPR by 3.20 to 9.23 points across 1% to 10% FPR.Each descriptive interval for these secondary metrics was entirely negative where the direction favored random sampling.

5 Discussion

The discussion finds that uncertainty sampling is more label-efficient under clean, low-budget conditions, but its advantage varies with noise structure, dataset, and evaluation metric. It recommends matched, budget-specific, repeated, multi-metric evaluation rather than assuming universal robustness.

  • Clean-label performance: Across all three datasets, uncertainty sampling used labels more efficiently than random sampling under clean labels, with the largest benefit when annotation budgets were scarce.The advantage narrowed as the budget grew, while random sampling remained a serious but weaker baseline in the tested clean low-budget regime.
  • Noise robustness: Difficulty-dependent noise reduced uncertainty sampling’s advantage more than RCN across a substantial rate range on Breast Cancer, but not on Banknote or MAGIC.The discussion attributes the dataset-specific pattern potentially to geometry, reference-model alignment, and the examples reached by each query path.
  • Noise robustness: After mean final acquired exposure was aligned, corrected analyses did not support a universal additional penalty from structured error location.The matched control weakens a one-mechanism explanation, but does not show that error location never matters or that exposure alone explains H2.
  • Evaluation metrics: On MAGIC, uncertainty sampling improved balanced accuracy while reducing average precision and fixed-FPR TPR, preventing the conclusion that it was simply better.These metrics capture different aspects of hard-threshold performance, ranking, and operating behavior, so evaluation should match the intended use.
  • Practical implications: For small tabular projects, uncertainty sampling is a reasonable clean-label baseline, but evaluation should compare budgets, retune with available labels, repeat paired runs, measure acquired errors, and report relevant operating points.When uncertain examples may receive unreliable labels, repeated labeling, expert escalation, or noise-aware query rules are more defensible than assuming universal failure or efficiency.

6 Limitations

The study’s conclusions are constrained by narrow datasets and models, synthetic noise and benchmark labels, limited noise and exposure matching, and uncertainty in secondary metric estimates and analysis registration.

  • Scope: Conclusions are limited to three binary tabular datasets, one logistic learner, and two basic query strategies.Deep models, multiclass problems, structured data, and alternative uncertainty or diversity methods may behave differently.
  • Noise construction: The noise processes are synthetic, difficulty is defined by an out-of-fold random-forest boundary, and benchmark labels may contain unknown errors.The reference boundary need not match the logistic learner’s boundary, so simulated difficulty is not observed annotator disagreement.
  • Design limits: Noise was tested only through 0.30, all point-estimate AULC crossings were right-censored, and matched exposure was assessed only at budget 120.Mean exposure was matched across seeds rather than exactly within each seed or checkpoint, limiting separation of exposure from location across the trajectory.
  • Evaluation and reporting: Fixed-FPR estimates were discrete at low FPR, secondary intervals and budget trajectories were descriptive, and the analysis plan was not publicly preregistered before computation.The MAGIC test splits contained 1,338 background examples, so a 1% empirical FPR limit allowed at most 13 false positives.

7 Conclusion

Uncertainty sampling was label-efficient across the tested datasets, but its apparent robustness to label noise was conditional rather than universal. Conclusions depended on dataset structure, annotation budget, noise process, and evaluation metric, requiring assessment across complete learning curves and realistic operating points.

  • 7 Conclusion: Uncertainty sampling improved clean-label balanced-accuracy label efficiency on all three datasets and retained a positive mean AULC advantage through 30% expected noise.The advantage was reduced more than RCN on Breast Cancer, but this pattern did not generalize to Banknote or MAGIC.
  • 7 Conclusion: Exposure-matched controls found no corrected evidence for a universal additional harm from structured error location, with two opposite-direction results on MAGIC.
  • 7 Conclusion: Uncertainty sampling can be label-efficient, but its apparent robustness depends on dataset structure, annotation budget, noise process, and evaluation metric.Active-learning claims should be evaluated across complete learning curves and realistic operating points rather than a single final score.

Reproducibility and Data Availability

The study used publicly available UCI datasets and provides its frozen configuration, deterministic implementation, audits, analysis plan, tests, and reporting code. Hash-bound artifacts support reproducibility across 12,800 trials and 271,600 retained budget rows.

  • Implementation and materials: Public UCI datasets and the complete reproducibility package are available, including frozen configuration, deterministic implementation, audits, analysis plan, tests, and reporting code.The materials are hosted at https://github.com/dev-juy/hard-cases-bad-labels.
  • Retained artifacts: 12,800 trials and 271,600 retained budget rows were preserved in hash-bound manifests covering the main, matched-control, extended-clean, and analysis artifacts.The manifests bind retained artifacts to their corresponding configurations.
  • Seed coverage: 100 seeds were used for each dataset in the main and matched-control components, while the extended clean-label component used 100 paired seeds on Breast Cancer Wisconsin only.The seed design is documented separately for the primary, matched-control, and extended clean-label analyses.

Deviations and Reporting Clarifications

The manuscript clarifies terminology while preserving the frozen analysis, hypotheses, trials, and statistical tests. Reporting revisions added no new scientific trials or post-outcome model variants.

  • Terminology clarification: “Bounded difficulty-dependent noise” replaces the internal identifier margin_dependent_massart_style because the simulated probability uses a random-forest boundary and is narrower than general Massart noise.The noise function, trials, hypotheses, and statistical tests were unchanged.
  • Reporting clarification: The H1–H4 numbering remains unchanged, while Table 2’s exposure range is a post-analysis descriptive summary of existing H4 output.The reporting clarification introduced no additional test, scientific trials, or post-outcome model variants.
Loading 2608.13601v1…