PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Classification-Risk-Optional Label Acquisition

F. Setoudehtanzangi, Geoffrey J. McLachlan

arXiv:2609.06873v1stat.MLcs.LG

TL;DR

With labels costly and selective, the paper asks how to allocate a limited budget to reduce multiclass zero–one risk. It combines conditional label information with local risk geometry; on Landsat, adaptive classification-risk targeting has lower mean error than the Fisher comparator across budgets, but does not uniformly outperform entropy or margin sampling.

  • Problem

    When features are observed broadly but labels are costly, it remains important to determine which observations to label, since posterior uncertainty alone does not capture their value for improving a classifier.

    Source: §1 · pp. 1–2

  • Method

    The paper combines Fisher information from acquired labels with local multiclass zero–one-risk curvature to allocate labels toward information aligned with boundary-relevant parameter directions.

    Source: §1 · p. 5, §11 · p. 37

  • Results

    On Landsat, adaptive classification-risk acquisition attained lower mean overall error than the ICC-targeted Fisher design across budgets, but did not uniformly outperform entropy or margin sampling.

    Source: §11 · p. 37

  • Takeaways & Limitations

    Classification value can differ from posterior ambiguity or global information because it weights label information by its local contribution to zero–one classification risk.

    Source: §11 · p. 37

    The theory is local and parametric and relies on correct model specification; under misspecification, it targets the working model’s risk geometry rather than the unknown population Bayes rule.

    Source: §10 · p. 36

Abstract

from arXiv · show

We study how a limited labeling budget should be allocated to minimize multiclass zero-one classification risk. We consider parametric classification problems in which features are observed for all sampling units while class labels can be acquired selectively. By combining the Fisher information supplied by an acquired label with the local geometry of multiclass excess risk, we derive an acquisition criterion that minimizes the leading asymptotic coefficient of expected multiclass excess risk. The resulting rule values a label according to how strongly its information is aligned with parameter directions that perturb the active Bayes decision boundary, rather than according to posterior uncertainty or global parameter information alone. We characterize the oracle acquisition design, establish its threshold structure, and derive face-specific and cost-sensitive extensions. An analytic example shows that posterior uncertainty and classification value can produce different, and even reversed, acquisition rankings. We further develop a two-stage adaptive procedure that attains the oracle leading-risk criterion under regularity conditions and provide explicit results for Gaussian discriminant analysis. Three-class QDA experiments illustrate the resulting acquisition geometry, while an application to the six-class Statlog Landsat Satellite data shows that classification-risk acquisition can differ materially from both uncertainty-based acquisition and the complete-classification-information comparator. The adaptive classification-risk design attains lower mean error than this Fisher comparator across the labeling budgets considered, although it does not uniformly outperform entropy or margin sampling and differences among the targeted strategies become small as the labeling budget increases.

1 Introduction

The paper asks how to allocate scarce labels for multiclass classification and derives a design that values information according to its effect on local zero–one risk and active Bayes boundaries.

1 Introduction

  • The oracle rule acquires labels when marginal classification value exceeds a budget-determined threshold, with possible randomization on the threshold set.The value combines conditional label information, information under the current design, and zero–one-risk importance; it also decomposes over active faces.Source: §1 · p. 4
  • An analytic example shows that observations with identical posterior uncertainty can have different classification values, even producing opposite acquisition rankings.The paper contrasts posterior ambiguity with directional label-information value and compares its risk-derived target with complete-classification information.Source: §1 · p. 4
  • Under stated regularity and consistency conditions, the two-stage plug-in design attains the oracle leading-risk coefficient asymptotically.For nonconstant labeling costs, the marginal criterion becomes classification value per unit cost.Source: §1 · pp. 4–5
  • The numerical study uses three-class Gaussian QDA, while the real-data application compares acquisition patterns on six-class Statlog Landsat Satellite data.The simulations include heterogeneous and near-linear configurations; the paper also reports a seven-class Dry Bean application in supplementary material.Source: §1 · p. 5
  • The paper derives a label-acquisition criterion from local second-order multiclass zero–one risk geometry, rather than proposing a generic trace-optimal design.Its target matrix weights parameter directions by their local contribution to movement of the active Bayes boundary.Source: §1 · pp. 3–5

2 Controlled label acquisition and classification risk

  • The expected leading excess-risk term depends on how acquired-label information reduces uncertainty in parameter directions that affect the active Bayes boundary.The risk-curvature matrix weights directions by their first-order effect on that boundary.Source: §2 · pp. 7–8

3 Classification-risk-optimal label acquisition

The oracle design allocates labels to minimize the leading excess classification-risk criterion under a labeling budget. Its threshold rule favors labels whose information aligns with parameter directions affecting the active Bayes boundary.

  • The oracle minimizes the leading excess-risk criterion by choosing where in feature space to acquire labels under the population budget.Source: §3 · p. 8
  • The oracle objective is convex and admits a solution under integrability, positive-semidefinite curvature, and uniform nonsingularity conditions.Source: §3 · p. 9
  • At the optimum, labels are acquired where their marginal classification value is largest, with randomization possible only on a value level set.Source: §3 · p. 10
  • A label’s value depends on whether its information aligns with estimation directions that affect the active Bayes boundary, not on total information alone.Source: §3 · p. 10
  • The classification-risk and complete-classification-information designs share a trace-inverse form but use different target matrices for their direction weights.Source: §3 · p. 11

4 Classification value and posterior uncertainty

Classification value combines label information with the parameter directions that affect the active Bayes boundary, so posterior uncertainty alone need not rank labels appropriately. The section also decomposes value by active face and extends the acquisition rule to heterogeneous costs.

4 Classification value and posterior uncertainty

  • In a regular binary logistic model with a linear Bayes boundary, equal posterior uncertainty can accompany different classification values and even opposite acquisition rankings.Source: §4 · pp. 12–13
  • In the example, entropy is determined by the coordinate normal to the boundary, while information about a coefficient that rotates the boundary raises value for observations with large |y2|.Source: §4 · pp. 12–13
  • Entropy and binary margin scores depend only on posterior probabilities, whereas classification value also reflects label information aligned with boundary-relevant parameter directions.Source: §4 · pp. 11, 13
  • The two rankings coincide in a restricted setting when classification value reduces to a scalar function of posterior uncertainty that is monotone in the chosen uncertainty measure.Source: §4 · p. 13

5 Face-specific classification value and acquisition costs

  • Face-specific values quantify how label information reduces leading risk associated with particular active Bayes faces, without requiring the observation to lie near that face.Source: §5 · p. 14
  • With heterogeneous costs, the oracle ranks labels by marginal classification value per unit cost, so a cheaper label can be preferred despite supplying less information.Source: §5 · pp. 15–16

6 Adaptive classification-risk-optimal acquisition

A two-stage adaptive procedure estimates the unknown quantities in the oracle criterion from a pilot sample and then acquires labels using the fitted design. Under stated regularity and uniform-convergence conditions, it attains the oracle leading-risk criterion without requiring a unique oracle design.

  • A pilot sample estimates the quantities in the oracle design, after which the fitted acquisition rule is used for the remaining observations.Source: §6 · pp. 16–17
  • Theorem 6.1 establishes convergence of the adaptive design’s criterion to the oracle value under uniform nonsingularity and stated approximation conditions.Source: §6 · p. 17
  • The result targets the minimum classification-risk coefficient, not a particular acquisition function; distinct designs may attain the same oracle value.Source: §6 · p. 17
  • With a vanishing pilot fraction and regular second-stage conditions, the fitted classifier retains the oracle first-order risk coefficient; the pilot may also join the final likelihood under standard asymptotic-linearity conditions.Source: §6 · p. 18

7 Gaussian discriminant specialization

The Gaussian discriminant specialization gives explicit information and risk-curvature quantities for evaluating classification-risk acquisition. Its face-based curvature representation retains the geometry of curved QDA boundaries and distinguishes classification value from posterior ambiguity.

  • For Gaussian discriminant models, the conditional information from a class label is the posterior covariance matrix of the class-score vector.Source: §7 · p. 20
  • The curvature representation weights boundary locations by class density and parameter contrast, with a denominator reflecting the local separation rate of the weighted class densities.Source: §7 · p. 21
  • Observations with nearly identical posterior probability vectors can receive substantially different classification values because their label information may target different boundary-relevant parameter combinations.Source: §7 · p. 22
  • Unequal class covariance matrices make QDA pairwise Bayes faces curved, retaining the full geometry of the classification-risk curvature.Source: §7 · pp. 19, 22

8 Numerical study

The simulations compare adaptive classification-risk acquisition with uncertainty, Fisher, random, and oracle designs in two three-class QDA geometries. The adaptive rule led implementable methods in the more heterogeneous setting, while its edge over Fisher narrowed in the near-linear control.

8.1 Simulation design

  • The study compares six acquisition strategies in two three-class Gaussian discriminant settings, fitting QDA after acquisition and evaluating error on a common 100,000-observation test sample.The adaptive rule uses pilot-estimated risk geometry; the oracle uses the true generating parameter and is a theoretical benchmark.Source: §8 · p. 22, §8.1 · p. 23

8.2 Scenario 1: heterogeneous QDA geometry

  • In Scenario 1, entropy clusters near ambiguous boundary regions, while classification-risk acquisition extends along active faces to select labels informative about their movement and deformation.The figure contrasts the regions, entropy, classification-value surface, and selected locations under a 20% budget.Source: §8.2 · p. 24, Figure 1 · p. 25
  • At 10% labels, adaptive risk-optimal error was 0.15138 versus Fisher 0.15156; at 20%, it was 0.15020 versus Fisher 0.15031.At 30%, targeted-method errors were between 0.14980 and 0.14992, versus 0.15043 for random acquisition.Source: §8.2 · p. 25

8.3 Scenario 2: weak covariance heterogeneity

  • In Scenario 2, near-linear boundaries bring entropy and classification value closer, though risk acquisition still spreads selected observations farther along active faces than entropy sampling.This setting reduces covariance heterogeneity relative to Scenario 1.Source: §8.3 · p. 26, Figure 2 · p. 27
  • At 10% labels in Scenario 2, adaptive error was 0.16193 versus Fisher 0.16205; at 20%, it was 0.16046 versus Fisher 0.16049.At 30%, adaptive and oracle risk-optimal mean errors were approximately 0.16002.Source: §8.3 · p. 27
  • Across scenarios, adaptive risk-optimal acquisition had the lowest implementable mean error in Scenario 1; targeted-rule differences narrowed with budget and were smaller versus Fisher in Scenario 2.Random acquisition had the largest mean error in both settings.Source: Table 1 · p. 28, §8.3 · pp. 28–29, Figure 3 · p. 29

9 Real-data application

In the Landsat benchmark, adaptive classification-risk acquisition had lower mean error than the ICC-targeted Fisher comparator at all budgets, but did not uniformly outperform entropy or margin. The discussion links label value to information aligned with active-boundary directions and states the framework’s scope.

9 Real-data application

  • The six-class benchmark uses four central-pixel spectral measurements; all 50 runs retain the same train/test samples but vary the random 5% pilot labels and resulting acquisitions.The benchmark has 4,435 training observations and 2,000 test observations.Source: §9 · pp. 29–30
  • Entropy concentrated on high-uncertainty regions, while classification value was more heterogeneous and produced different label selections; principal components only visualize the full four-dimensional QDA analysis.Figure 4 shows the two selection patterns at a 20% budget.Source: Figure 4 · p. 31, §9 · pp. 31–32
  • Across all three budgets, adaptive classification-risk acquisition had lower mean overall error than Fisher, but did not uniformly beat entropy or margin, and differences narrowed at 30%.The 30% mean errors were 0.15657 for adaptive acquisition and 0.15716 for Fisher; targeted methods were close at that budget.Source: Table 2 · p. 32, §9 · pp. 33–34
  • At 20%, class-specific differences between adaptive and Fisher acquisition were heterogeneous; the authors treat these results as descriptive, not evidence of improvement for every class.Adaptive acquisition had lower class-specific error for some classes, while Fisher performed better for others.Source: §9 · pp. 33–34

10 Discussion

  • In heterogeneous QDA, adaptive acquisition had the smallest mean error among implementable procedures, while more similar covariance matrices brought Fisher and risk-targeted designs closer.Differences among targeted methods also decreased as the labeling budget increased; the authors caution against interpreting the results as universal dominance.Source: §10 · p. 35
  • The oracle interpretation is local and parametric and assumes a correctly specified working classifier; under misspecification, the criterion targets that model’s risk geometry rather than the unknown Bayes rule.Semiparametric or nonparametric extensions would need to control estimation error and decision-boundary stability.Source: §10 · p. 36

11 Conclusion

  • The oracle acquisition problem is convex with a threshold rule favoring the largest marginal classification values; extensions cover active faces, labeling costs, and adaptive attainment under regularity conditions.The Gaussian discriminant specialization gives explicit information and boundary-curvature expressions for multiclass LDA and QDA.Source: §11 · p. 37
  • A label’s value depends on whether its information acts in parameter directions that influence the active multiclass decision boundary, not on posterior uncertainty or total Fisher information alone.This is the paper’s stated connection between label acquisition, statistical information, and local classification-risk geometry.Source: §11 · p. 37

Supplementary Material for Classification-Risk-Optimal Label Acquisition

The supplement proves information and classification-risk results for controlled acquisition. Its proofs decompose the observed-data score and apply a local excess-risk expansion under regularity conditions.

S1 Information and classification-risk results under controlled acquisition

Proof of Proposition 2.1

  • Under controlled acquisition, the observed-data score splits into feature and acquired-label terms that are orthogonal in expectation.Source: Proof of Proposition 2.1 · p. 40

Proof of Proposition 2.2

  • The proof of Proposition 2.2 uses regular asymptotic normality and a local multiclass excess-risk expansion.A moment condition supplies uniform integrability, allowing convergence in distribution to yield the stated result.Source: Proof of Proposition 2.2 · pp. 41–42

S2 Proof of the oracle acquisition result

The proof establishes the oracle design through convexity, first-order conditions, and a threshold characterization. The optimal allocation prioritizes feature values with the largest marginal values.

Budget saturation

  • Additional labeling cannot increase the design criterion, so an oracle design can be chosen to use the full labeling budget.Source: Budget saturation · pp. 42–43

Convexity and existence

  • The criterion is convex when the risk-curvature matrix is positive semidefinite, and the proof establishes existence under uniform nonsingularity.Source: Convexity and existence · pp. 43–44

Directional derivative

  • The proof evaluates feasible design perturbations through their induced change in the information matrix.It defines that change as B_h = E{h(Y)J(Y)} and writes I(a + th) = I(a) + tB_h.Source: Directional derivative · p. 44

Equivalence condition

  • Convexity makes the first-order condition equivalent to global minimization, and the optimal design has a linear-problem interpretation conditional on its marginal-value function.Source: Equivalence condition · p. 44

Threshold characterization

  • The oracle allocation gives acquisition probability first to feature values with the largest marginal values, with zero acquisition below the threshold.Allocations at the threshold can be chosen to meet the budget; any feasible design satisfying the threshold condition is globally optimal.Source: Threshold characterization · p. 45

S3 Posterior uncertainty and classification value

The analytic example shows that posterior uncertainty and classification value can rank feature values differently. It also establishes a strict reversal between their rankings in a constructed case.

S4 Face-specific and cost-sensitive acquisition results

The proofs extend the design argument to face-specific and cost-sensitive acquisition results. For cost-sensitive acquisition, the allocation ranks locations by marginal value relative to cost.

Proof of Proposition 5.1

  • For an admissible perturbation, the proof establishes the directional derivative; as in Supplementary Section S2, this proves equation (5.5) and completes the proof of Proposition 5.1.Source: Proof of Proposition 5.1 · pp. 50–51

Proof of Corollary 5.2

  • An oracle cost-sensitive design can be chosen to satisfy the budget with equality.Source: Proof of Corollary 5.2 · p. 51
  • The cost-sensitive linear problem allocates acquisition capacity first to locations with the largest values of χ⋆.A cost-preserving transfer from a lower-value location to a higher-value one would improve the objective.Source: Proof of Corollary 5.2 · p. 52

S5 Adaptive acquisition and oracle classification risk

The section proves convergence of the plug-in acquisition design to the oracle population criterion and establishes a first-order excess-risk limit for the adaptive procedure. The proof relies on uniform convergence and stated regularity conditions.

S5 Adaptive acquisition and oracle classification risk

Uniform convergence of the plug-in criterion

  • The plug-in criterion argument uses uniform control over acquisition levels, bounded inverse information matrices, and the resolvent identity.The text states that substitution into (S5.1) establishes the claim.Source: Uniform convergence of the plug-in… · p. 53

Proof of Theorem 6.1

  • Theorem 6.1's proof compares a fitted-problem minimizer with an oracle design and establishes population feasibility and criterion optimality in the limit.The argument also covers an op(1)-approximate minimizer.Source: Proof of Theorem 6.1 · pp. 54–55

Proof of Theorem 6.2

  • Under the stated conditional regularity and efficiency assumptions, the second-stage estimator is asymptotically centered normal, and the first-order limit is unchanged when Nn is replaced by n.Uniform integrability of scaled excess risks supports convergence of expectations.Source: Proof of Theorem 6.2 · pp. 55–56

S6 Gaussian discriminant calculations

The section derives Gaussian discriminant information and boundary-curvature expressions used in Section 7. It supplies class-score, pairwise-face, and acquisition-value calculations.

S6 Gaussian discriminant calculations

Conditional label information

Explicit Gaussian class scores

  • Under the baseline-logit parameterization, the Gaussian QDA class score contains a log-determinant term and a quadratic covariance-weighted distance term.The section proceeds by direct differentiation and treats the covariance matrix separately.Source: Explicit Gaussian class scores · p. 57

Proof of Proposition 7.1

S7 Additional real-data application: Dry Bean data

In the Dry Bean application, adaptive classification-risk acquisition performed best at the 10% labeling budget, but not at larger budgets. The visualization also shows that entropy and classification value can select different observations.

Data and preprocessing

  • The 13,543-observation analysis used seven bean varieties and retained five principal components, which explained approximately 97.8% of predictor variation.The original data had 13,611 observations and 16 predictors; 68 exact duplicates were removed.Source: S7 Additional real-data application:… · p. 60

Model scope

  • The Gaussian QDA model was a working approximation, not an assumption of an exactly Gaussian-mixture population; the application therefore examines behavior under possible misspecification..Source: S7 Additional real-data application:… · p. 60

Acquisition geometry

  • Entropy sampling concentrates broadly in uncertain regions, while classification value varies within them and leads the two rules to select different subsets.Figure S1 uses the first two principal components only for display; decisions use the full five-dimensional model.Source: S7 Additional real-data application:… · p. 62, Figure S1 · p. 62

Results

Paired comparisons

  • At the 10% budget, adaptive reduced mean test error versus random by 0.0106 and versus entropy by 0.0030, with both paired 95% Monte Carlo intervals excluding zero.Its improvements over margin and Fisher were smaller, about 0.0012 and 0.0009, with intervals including zero.Source: S7 Additional real-data application:… · p. 64

Results

  • At 20%, Fisher mean error was 0.09588 versus 0.09655 for adaptive, a paired difference of approximately 6.7 × 10−4 in Fisher's favor.The adaptive rule's mean error reductions versus random, entropy, and margin were approximately 0.0020, 0.0043, and 0.0041, respectively.Source: S7 Additional real-data application:… · p. 64
  • At 30%, random mean error was 0.09178, compared with 0.09390 for Fisher and 0.09437 for adaptive; adaptive remained below entropy and margin.The adaptive–Fisher difference was 0.00047 and was not clearly distinguishable from zero in the paired comparison.Source: S7 Additional real-data application:… · p. 64
Loading 2609.06873v1…