PaperlayerView exact revision on arXiv

Source-linked AI summary

Not All Mislabels Are Equal: Efficient Task-Aware Auditing with Confidence Sequences

Fangyuan Lin, Noah Krever, Jakub Černý, Lily Xu

arXiv:2610.33262v1stat.MEcs.LGmath.STstat.APstat.MLstat.TH

TL;DR

Imperfect categorical labels can distort downstream scores, yet expert verification is costly and errors on different labels can have different effects. ATLAS combines task-aware adaptive auditing with anytime-valid inference, reducing audit counts for linear and nonlinear score targets while retaining finite-sample guarantees. A Bayesian alternative offers greater efficiency when distribution-free auditing is too costly, at the cost of assumption-free validity.

  • Problem

    The paper addresses how to certify downstream quantities computed from imperfect categorical labels and choose which records to audit when expert verification is costly and correction impacts differ.

    Source: §1 · pp. 1–2

  • Method

    ATLAS represents corrections with a confusion matrix and combines task-aware adaptive expert review with anytime-valid inference, using linear-envelope representations for suitable nonlinear scores.

    Source: §1 · p. 3, §8 · p. 10

  • Results

    PAS uses 24% fewer audits in Kenya and 11% fewer on CIFAR-10N for mislabel-rate certification, and 36.4% fewer audits for nonlinear Hill–Shannon certification.

    Source: §4.1 · p. 6, §5.1 · p. 8

  • Takeaways & Limitations

    Task-aware auditing can reduce costly expert reviews while retaining finite-sample, anytime-valid guarantees for supported downstream scores.

    Source: §1 · p. 3, §8 · p. 10

    Prospective use must prespecify the eligible label vocabulary, preserve the randomized design, and account for expert error when verified labels are not definitive.

    Source: ETHICS STATEMENT · p. 11

Abstract

from arXiv · show

We study efficient, adaptive auditing of categorical labels in a fixed dataset, motivated by biodiversity-credit programs in which expert verification of species records is costly and downstream scores (e.g., payment) are more sensitive to errors on certain labels than on others. Our proposed ATLAS framework couples anytime-valid inference with a task-aware adaptive auditing policy. Representing label corrections through a confusion matrix, we prioritize which data points to inspect next based on both label uncertainty and the influence of potential label corrections on the downstream score function. For linear scores such as the mislabel rate, our priority-aware approach certifies the overall label error rate to $\pm 2$ percentage points with 24% and 11% fewer audits than random sampling on biodiversity data from Kenya and a widely used CIFAR-10N benchmark, respectively. For nonlinear scores such as the Hill-Shannon diversity, we exploit a linear-envelope representation that reduces inference to a family of linear-score audits; priority-aware sampling reaches the target precision with 36% fewer audits. When distribution-free guarantees are prohibitively costly, we develop a model-based Bayesian alternative that trades assumption-free validity for substantially fewer audits. Our biodiversity case study highlights a gap between statistical validity and practical feasibility: generic methods such as Serfling's bound and existing confidence-sequence procedures are rigorous but can require prohibitively many expert reviews in practice. Together, these results show task-aware auditing can significantly reduce the number of expert audits needed while providing finite-sample, anytime-valid guarantees.

1 INTRODUCTION

The paper asks how to certify downstream quantities from imperfect labels while limiting costly expert review, and how to choose records efficiently. ATLAS combines adaptive auditing with anytime-valid inference, accounting for how corrections affect downstream scores.

Motivation: Auditing for biodiversity credits

  • Kenya’s submitted labels imply pooled Hill–Shannon diversity of 8.16, versus 4.94 after expert validation, illustrating score-sensitive label errors.The direction of an error matters: mislabeling cattle as lion can inflate estimated richness and diversity, whereas confusing common related species may have little effect.Source: §1 · p. 2

Contributions

  • ATLAS combines e-process inference and adaptive expert review to certify downstream quantities while prioritizing corrections that matter most to the score.Its framework is designed for fixed categorical datasets and retains finite-sample, anytime-valid guarantees.Source: §1 · p. 3
  • For nonlinear scores such as Hill–Shannon diversity, linear-envelope representations reduce inference to a family of linear-score audits.The resulting score-specific confidence sequences retain finite-sample, anytime-valid guarantees.Source: §1 · p. 3
  • When distribution-free guarantees require impractically many audits, a Bayesian confusion-matrix model offers an alternative, trading assumption-free validity for greater efficiency.Source: §1 · p. 3

2 PROBLEM FORMULATION

The paper formulates auditing as finite-population inference over corrections to submitted categorical labels. It tracks label-pair corrections and metadata-defined groups so audits can target uncertainty relevant to a downstream score.

Dataset and labels

  • Each record has a known submitted label and an unknown verified label drawn from the full normalized label vocabulary.Source: §2 · p. 3

Confusion matrix

  • The unknown finite-population confusion matrix counts records by submitted label and verified label, with its diagonal representing correctly labeled records.A scalar mislabel rate records only how often labels are wrong, not which classes are involved.Source: §2 · p. 3

Score functions

  • A score function maps the submitted-to-verified confusion matrix to a scalar outcome, and the target is the score for this particular dataset rather than an underlying population parameter.Examples include error rates, model-evaluation metrics, and biodiversity scores.Source: §2 · p. 3

Strata: splitting rows by metadata

  • Strata group records by submitted label and metadata; when metadata is absent, each submitted-label row is its own stratum.Known stratum sizes let auditing focus on metadata-defined subgroups where correction behavior varies or uncertainty remains.Source: §2 · p. 3

3 CONFIDENCE SEQUENCES FOR AUDITING LINEAR SCORE FUNCTIONS

For linear scores of a fixed dataset, the method turns each sequential audit into an unbiased score observation and uses e-processes to form an anytime-valid confidence sequence. The auditor can monitor precision and stop adaptively without losing coverage validity.

Linear scores

  • For linear scores, each record’s contribution is fixed by its submitted and verified labels, and the population score is the average contribution over the confusion matrix.For the mislabel rate, the contribution matrix is binary, marking whether the submitted and verified labels differ.Source: §3 · p. 4

A starting point: Serfling’s inequality

  • Serfling’s bound requires 4,154 reviews for a ±2-percentage-point error-rate interval, while the Wald interval requires 1,481 but offers only an asymptotic approximation.The section also states that the e-process construction extends to ratios of linear scores, including recall and F1 score.Source: §3 · pp. 4–5

Constructing an unbiased audit observation

  • The auditor constructs an unbiased observation of the downstream score by correcting a prediction with an inverse-probability-weighted audit residual.The prediction term acts as a control variate; better predictions reduce variance, while unbiasedness holds regardless of prediction quality.Source: §3 · pp. 4–5
  • After fixing predictable sampling probabilities and predicted verified-label distributions, the auditor samples a stratum and then a remaining record uniformly within it.An unaudited record in stratum g is selected with probability πg,t/Mg,t.Source: §3 · p. 4

E-processes

  • Two e-processes test each candidate score value, and retaining values not rejected by either process yields a confidence sequence with simultaneous coverage over audit times.The process can stop when the desired precision is reached.Source: §3 · pp. 4–5

4 PRIORITY-AWARE SAMPLING

Priority-aware sampling allocates audits using both the remaining size of each stratum and uncertainty in its score contributions. On two datasets, it reaches the mislabel-rate precision target with fewer audits than simple random sampling.

4 PRIORITY-AWARE SAMPLING

  • With exact predictions, the variance-minimizing allocation assigns probability in proportion to each stratum’s remaining records times its contribution uncertainty, subject to probability floors.Source: §4 · p. 5
  • The practical allocation mixes the priority probabilities with sampling proportional to remaining stratum size, preserving exploration while approximating the variance-minimizing allocation.Source: §4 · p. 6

4.1 EXPERIMENTAL RESULTS: CERTIFYING THE MISLABEL RATE

  • 24% fewer reviews in Kenya (656 vs 864) and 11% fewer on CIFAR-10N (768 vs 862) let PAS certify the mislabel rate to ±2 percentage points.PAS retains the underlying confidence sequence’s anytime-valid guarantee; Table 1 reports median stopping times over 20 audit paths.Source: §4.1 · p. 6, Table 1 · p. 6
  • PAS reaches the two-percentage-point target earlier than the prior–posterior-ratio method, the Wald interval, and Serfling’s inequality on both datasets.These benchmarks differ in validity: the prior–posterior-ratio method is anytime-valid, Wald is asymptotic, and Serfling is finite-sample.Source: §4.1 · p. 6

5 AUDITING NONLINEAR SCORES VIA LINEAR ENVELOPES

The paper extends anytime-valid auditing to concave nonlinear scores by bounding them with linear scores. Across Kenya and CIFAR-10N, ATLAS-PAS reaches specified relative-precision targets with fewer audits than ATLAS-SRS.

5 AUDITING NONLINEAR SCORES VIA LINEAR ENVELOPES

  • For concave scores, a predictable tangent linear score gives an upper bound, while a lower bound rules out feasible compositions with entropy at most a candidate value.The lower-bound construction uses a shared predictable stake and convex optimization to certify the minimum across candidate compositions.Source: §5 · pp. 6–8

5.1 EXPERIMENTAL RESULTS: CERTIFYING NONLINEAR SCORES

  • 36.4% fewer audits: ATLAS-PAS reaches a 5% relative margin-of-error target for pooled Hill–Shannon diversity after 1,622 audits, versus 2,550 under ATLAS-SRS.Every stored interval contained the fixed-population truth.Source: Figure 3 · p. 8, §5.1 · p. 8
  • 44% fewer audits: ATLAS-PAS reaches a 5% relative margin-of-error target for worst-three precision after 3,944 audits, versus 7,087 under ATLAS-SRS.Worst-three precision is evaluated on CIFAR-10N.Source: Figure 3 · p. 8, §5.1 · p. 8
  • 44% fewer audits: ATLAS-PAS reaches a 25% relative half-width target for worst-three taxonomic risk after 3,078 audits, versus 5,483 under ATLAS-SRS.At 10,000 audits, PAS intervals are 19% narrower for this score.Source: Figure 3 · p. 8, §5.1 · p. 8

6 A BAYESIAN, MODEL-BASED ALTERNATIVE FOR EFFICIENT AUDITING

The Bayesian alternative shares information across related labels and strata to improve auditing efficiency when distribution-free procedures require impractically many audits. It uses posterior uncertainty over corrected datasets to assess a nonlinear score.

6 A BAYESIAN, MODEL-BASED ALTERNATIVE FOR EFFICIENT AUDITING

  • The model-based alternative shares information across related labels and strata, trading robustness to model specification for greater statistical efficiency.The same task-aware auditing framework accommodates both model-based and distribution-free settings.Source: §6 · p. 9
  • A hierarchical Dirichlet–multinomial model combines label-level and stratum-level correction distributions, with audited local counts increasingly dominating the shared prior.Concentration parameters control how strongly information is shared across labels and strata.Source: §6 · p. 9
  • After audit 100 and every 25 audits thereafter, posterior draws impute remaining labels in 400 corrected datasets, and the audit stops when the 95.5% interval meets a 5% relative-accuracy target.The reported interval uses the 2.25th and 97.75th empirical quantiles of the nonlinear score.Source: §6 · p. 9
  • On the Kenyan dataset, approximate Bayesian PAS required a median of 250 audits versus 500 under random sampling; at stopping, PAS and SRS credible intervals contained the truth on 96 of 100 paths.Source: §6 · p. 9

7 RELATED WORK

Related work provides anytime-valid inference for finite populations, adaptive weighted-mean audits, entropy, and active evaluation. This paper applies these ideas to verifying existing categorical labels according to their effects on downstream scores.

7 RELATED WORK

  • Confidence sequences remain valid over time and permit adaptive stopping, while finite-population methods address sampling without replacement and adaptive nonuniform sampling.Prior work includes confidence sequences for weighted means using importance-weighted betting martingales and control variates.Source: §7 · pp. 9–10
  • Unlike weighted-mean methods whose priority depends on scalar values, this work targets structured categorical corrections and derives a conditional variance to motivate allocation.The sampling objective is tied to downstream score effects.Source: §7 · p. 10
  • For entropy, the construction combines a linear-envelope upper bound with a certified convex-profile lower bound to test entropy levels directly.It incorporates predictions of unaudited labels and is designed to reduce conservatism from projecting a generic joint confidence region.Source: §7 · p. 10
  • Active evaluation labels test points to estimate predictive performance, whereas this work verifies already-labeled records and prioritizes corrections by their downstream-score effects.Those effects can be nonlinear.Source: §7 · p. 10

8 CONCLUSION

ATLAS uses task-aware audit allocation to obtain narrow confidence sequences for downstream scores while retaining finite-sample, distribution-free, anytime-valid guarantees. The conclusion qualifies these gains by score structure, data heterogeneity, and possible expert-label error.

8 CONCLUSION

  • Task-aware auditing leverages score structure to reduce expert audits while retaining finite-sample, distribution-free, anytime-valid guarantees.The conclusion contrasts this practical aim with rigorous methods such as Serfling’s inequality that may require prohibitively many audits.Source: §8 · p. 10
  • ATLAS represents corrections with a confusion matrix and combines adaptive audit allocation with inverse-probability-weighted inference.It directs audits toward unresolved corrections that matter most to the downstream task.Source: §8 · p. 10
  • The guarantees cover linear scores and nonlinear scores admitting suitable linear-envelope representations; direct sequential estimators for nonlinear scores such as entropy remain an open direction.Another open question is how closely distribution-free procedures can approach the empirical efficiency of the model-based Bayesian alternative.Source: §8 · p. 10
  • Priority-aware gains are limited on highly homogeneous data, and the analysis treats expert-verified labels as ground truth even though those labels may be erroneous.Further extensions include heterogeneous audit costs, multiple auditors, and nonstationary confusion patterns.Source: §8 · pp. 10–11

REPRODUCIBILITY STATEMENT

The supplementary materials document implementation and experimental resources, ethical boundaries, biodiversity-score scope, notation, and algorithm procedures.

REPRODUCIBILITY STATEMENT

  • The supplementary repository includes implementation, frozen seeds, saved results, figure-generation scripts, and public CIFAR-10N data; Kenyan biodiversity data are confidential and not redistributed.Anonymized aggregate results and corresponding experimental code are included.Source: REPRODUCIBILITY STATEMENT · p. 11

ETHICS STATEMENT

  • The biodiversity study is a retrospective replay, not an independent validation of credits or payments; prospective use should prespecify the label vocabulary, preserve randomized design, and account for expert error.CIFAR-10N annotations are treated as a designated comparison label, not proof that either source is infallible.Source: ETHICS STATEMENT · p. 11

AI USE STATEMENT

  • Generative AI tools assisted with proof checking, literature searches, experiment and figure-generation code, and high-level manuscript polishing.Source: AI USE STATEMENT · p. 11

A AUDITS FOR BIODIVERSITY CREDIT PROGRAMS

  • The biodiversity application uses sensor records of wildlife observations labeled by species, and considers the species-specific pillars of a multimetric that determines landowner payments.The multimetric also includes two habitat pillars, which are outside the considered scope.Source: A AUDITS FOR BIODIVERSITY CREDIT… · p. 14

B NOTATION

  • Table 3 collects core notation used in the main text; symbols used only in the appendix are defined where first introduced.Source: Table 3 · p. 15

C ALGORITHM PSEUDOCODE

  • Algorithm 1 computes predictions, priorities, sampling probabilities, and stakes before each verified label is observed; prediction quality affects efficiency but not confidence-sequence validity.Verified strata are removed and the sampling distribution is renormalized over the remaining records.Source: C ALGORITHM PSEUDOCODE · p. 16
  • For entropy, the upper endpoint uses a predictable linear cross-entropy direction, while the lower endpoint requires certified convex optimization over compatible compositions.The interval is valid only when the numerical lower-bound optimization returns a certified bound.Source: C ALGORITHM PSEUDOCODE · p. 16

D EXPERIMENT SETTINGS AND ADDITIONAL RESULTS

The experiments specify frozen auditing settings for linear and nonlinear scores on biodiversity and CIFAR-10N data. They cover distribution-free and Bayesian diversity audits, plus CIFAR-10N worst-three scores.

D EXPERIMENT SETTINGS AND ADDITIONAL RESULTS

  • The error-rate experiments use nominal 95.5% confidence, with PAS and SRS sharing a confidence sequence and differing only in predictable sampling probabilities.Records are sampled uniformly within selected strata, and paired policies use identical randomized within-stratum orders.Source: D EXPERIMENT SETTINGS AND ADDITIONAL… · p. 18

D.1.1 LINEAR SCORE: MISLABEL RATE

  • Kenya’s mislabel-rate settings use submitted labels as 46 strata, a two-stage shrinkage error model, and PAS with a 70% predicted Neyman-allocation component.PAS mixes that component with 30% remaining-record SRS; the experiments use 20 paired paths.Source: §D.1.1 · p. 18

D.1.2 NONLINEAR SCORE: PILLAR 2 DIVERSITY

  • Hill–Shannon diversity is the effective number of species implied by corrected labels, and its uncertainty can be especially consequential for rare or poorly audited species.The distribution-free experiment uses 300 submitted-label-by-camera-station strata; PAS allocation combines proportional, range-control, and entropy-direction Neyman components.Source: §D.1.2 · pp. 18–19

D.1.3 BAYESIAN SETTINGS

  • The Bayesian diversity comparison holds the hierarchical Dirichlet–multinomial model, posterior computation, credible interval, and stopping rule fixed, varying only audit allocation.Posterior intervals are evaluated every 25 audits beginning at 100, using 400 draws; PAS allocation inflates predicted uncertainty in lightly audited strata.Source: §D.1.3 · p. 19

D.2.1 LINEAR SCORE: MISLABEL RATE

  • CIFAR-10N’s mislabel-rate experiment forms 40 strata from submitted class and auxiliary-annotation agreement patterns, then selects a frozen PAS policy on development paths.The selected policy uses prior strength 50, effective horizon 1,200, and a 10% SRS component; confirmation uses 20 fresh paired paths.Source: §D.2.1 · pp. 19–20

D.2.2 NONLINEAR SCORE: WORST-THREE SCORES

  • Worst-three precision averages the three least precise submitted classes, while worst-three taxonomic risk averages the three highest-risk classes using normalized WordNet distances.The reported stopping targets correspond to relative half-widths of at most 5% for precision and 25% for taxonomic risk.Source: §D.2.2 · pp. 20–21

D.3 SYNTHETIC DATA: A PRIORITY MECHANISM STUDY

The synthetic study isolates how priority-aware sampling benefits from concentrating audits on records whose verified labels can change the score. The accompanying sections describe sampling comparators, computational setup, and linear-score prediction.

D.3 SYNTHETIC DATA: A PRIORITY MECHANISM STUDY

  • In the synthetic population, only four of 20 submitted-label rows have uncertain score contributions; PAS gives no audits to constant-contribution rows, unlike SRS’s uniform allocation.The population has 5,000 records, and the target linear risk is 0.30.Source: §D.3 · pp. 21–22
  • The scaling screen varies total rows from 8 to 48 while holding four score-variable rows fixed, testing whether PAS’s advantage grows with score-invariant nuisance rows.Confirmation fixes 20 rows and uses 100 fresh paths.Source: §D.3 · p. 22

D.4 RANDOM-SAMPLING COMPARATORS

  • Wald and Serfling provide fixed-sample benchmark sizes, not anytime-valid stopping rules; Wald uses oracle variance, while Serfling is a finite-population bound.The prior–posterior-ratio comparator is anytime-valid but does not use priority-aware allocation.Source: §D.4 · p. 22

E COMPUTATIONAL ENVIRONMENT

  • The experiments ran CPU-only on a local machine with 15 CPU cores and 24 GB of unified memory.Independent score-policy-seed tasks were distributed across four local worker processes.Source: E COMPUTATIONAL ENVIRONMENT · p. 22

F LINEAR SCORES

  • The two-level prediction estimator uses audits available before each review, shrinking stratum predictions toward row-level verified-label frequencies and rows toward a prior centered on submitted labels.For Kenya, β = 0.8, κr = 2, and κc = 1 were fixed before auditing and not tuned.Source: F LINEAR SCORES · p. 23

F.1 PROOFS FOR SECTION 3

The linear-score proofs establish valid confidence sequences under adaptive, without-replacement auditing and explain how prediction and stake choices affect their width. The resulting width depends on observed variance rather than a worst-case bound.

F.1 PROOFS FOR SECTION 3

  • The sampling probabilities cancel in the estimator’s conditional expectation, preserving unbiasedness for the finite-population target under adaptive stratum selection.The proof conditions on the past audits and selected stratum, with a uniformly sampled unaudited record within that stratum.Source: §F.1 · pp. 23–24
  • The allocation minimizing the stated variance objective is proportional to stratum size times predicted standard deviation when probability floors are absent.The optimization is convex when floors are imposed.Source: §F.1 · p. 24
  • The predictable plug-in stake replaces unknown variance with an estimate from past reviews and uses a planned horizon to choose stake size.A cap keeps early stakes small before variance is observed; the experiments use a horizon of 800 reviews.Source: §F.1 · pp. 24–25
  • The confidence sequence’s width is governed by observed variance, unlike Serfling’s width, which uses the maximum possible variance of 1/4.The raw error-indicator variance is reported as 0.147 and is described as far smaller after predictions learn the rows.Source: §F.1 · p. 25

G ENTROPY AND LINEAR ENVELOPES: CONSTRUCTION AND PROOFS

The entropy construction builds anytime-valid upper and lower bounds for corrected-composition entropy, then converts the interval to a Hill–Shannon diversity report. Its computation uses a convex profile and a priority-aware allocation, with separate constructions for other scores.

G.1 TARGET AND GUARANTEE

  • The target is an anytime-valid entropy interval whose transformed Hill–Shannon diversity report has at most 5% relative error when the log-interval width is at most log(21/19).The entropy error budget is split between lower- and upper-end coverage.Source: §G.1 · p. 25

G.2 THE REFERENCE COMPOSITION AND THE OBSERVATION FROM EACH AUDIT

  • Audits yield cross-entropy observations whose conditional means support the entropy bounds; the predictable reference composition affects interval width through its divergence from the true composition.The construction mixes in a uniform distribution to keep contributions finite, including for labels not yet seen.Source: §G.2 · p. 26

G.3 THE UPPER END

  • The upper endpoint is an anytime-valid confidence bound for entropy under the stated predictable-stake construction.Its coverage holds for every labeling.Source: §G.3 · p. 27

G.4 THE LOWER END

  • The lower endpoint is also valid for every labeling, and together the two endpoints form the entropy interval.The lower-end proof uses a predictable stake satisfying the stated cap condition.Source: §G.4 · p. 28

G.5 COMPUTING THE LOWER END: THE CONVEX PROFILE

  • The lower-end profile is convex, allowing a supporting-plane lower bound to certify candidate entropy values without invalidating coverage.An inexact optimizer can only widen the interval by losing evidence, not validity.Source: §G.5 · pp. 28–29

G.6 ALLOCATION AND CROSS-CAPPED STAKES

  • Priority-aware allocation uses the entropy influence direction, while cross-capping both endpoints to the previous interval preserves the stated interval guarantee.The allocation assigns every nonempty stratum positive probability.Source: §G.6 · p. 29

H SCORES BEYOND THE BIODIVERSITY PILLARS

  • For ratio scores such as recall, F1, and balanced accuracy, candidate values induce linear-score audits; Shannon entropy is outside the bounded-curvature approach because its second derivative is unbounded near zero.For support-based richness, certifying absence can require at least (1 − α)N expected reviews, whereas presence is certified by one verified record.Source: H SCORES BEYOND THE BIODIVERSITY PILLARS · pp. 29–30

Related papers

Loading 2610.33262v1…