PaperlayerView exact revision on arXiv

Source-linked AI summary

When Does Data Value Reduce to Class Balance? A Coverage View of Per-Point Data Valuation

Yingjie Bai, Weiming Zhi

arXiv:2610.34801v1cs.LGstat.ML

TL;DR

Fixed per-point top-k scores can waste a data-selection budget on redundant modes because a point’s marginal value depends on what has already been selected. The paper analyzes this gap and compares balancing scores with learner-geometry coverage. Its results support choosing between them according to whether selection labels identify the learner’s modes.

  • Problem

    Fixed per-point scores are widely used to select training data, but the paper asks whether top-k-by-score is suitable when a binding budget, redundant representation, and non-interpolable task make data selection consequential.

    Source: §1 · p. 1

  • Method

    The paper models point value as a selected-set context function, analyzes its coverage gap in latent-mode settings, and tests balance-versus-coverage choices across synthetic and real experiments.

    Source: §1 · pp. 2–3, §2 · p. 3

  • Results

    Across synthetic and real ViT experiments, when modes coincide with labels, balancing the same scores recovers missing coverage and can outperform label-free coverage.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Balance scores when available labels name learner modes; otherwise use explicit learner-geometry coverage, which has no handle on modes absent from the representation.

    Source: §6 · p. 18

    When point quality varies independently of coverage, pure coverage may select unique but low-quality outliers, while quality-filter-then-coverage performs best.

    Source: §6 · p. 18

Abstract

from arXiv · show

Per-point data-valuation scores -- Data Shapley, influence, leverage -- are routinely used to select training data: keep the points with the top $k$ scores. We argue that top-$k$ by a fixed, mode-blind per-point score is the wrong selection rule precisely in the regime where data selection matters. The reason is that a point's value is not a fixed number but depends on what else has been selected -- a near-duplicate of an already-chosen point adds almost nothing, however high its score -- and a per-point scalar cannot express this interaction. The theory quantifies this failure exactly: in an equal latent-mode model, fixed top-$k$ selection incurs a closed-form excess risk over adaptive coverage because it repeatedly spends budget on already-covered modes. The gap is largest when the budget is tight, the learner representation is redundant, and missing a mode is costly. The practical rule is simple: use balance when the labels available at selection identify the learner's modes, and use explicit coverage when they do not. On standard classification tasks with strong pretrained features, class-balancing the same per-point scores recovers the missing coverage and matches or outperforms generic set coverage across our datasets. Without labels, label-free coverage improves accuracy by up to +0.20 while raw top-$k$ can fall below random; when labels are coarser than the modes, coverage within coarse strata performs best. We turn this distinction into a pre-specified diagnostic and validate it across both favorable and stress-test settings. The limits are part of the claim: the advantage is representation-relative. Coverage fails when the relevant modes are not expressed in the learner geometry, as with spurious subgroups, or when data quality varies independently of coverage -- exactly as the theory predicts.

1 Introduction

Per-point scores are widely used to select training data, but a fixed top-k rule can waste a binding budget on redundant modes. The paper analyzes this failure and proposes choosing between label balancing and learner-geometry coverage according to whether selection labels identify the modes.

1 Introduction

  • At k = C = 10, Shapley covers 6.0 modes and leverage 6.7, versus greedy coverage’s 9.7; the synthetic binding-budget gap is 0.895 vs. 0.637 (+0.258).The gap matters under a binding budget, redundant learner geometry, and a hard task.Source: §1 · p. 2, Figure 1 · p. 2
  • A point’s value depends on the selected set: a same-mode near-duplicate adds little after that mode is covered, which a fixed scalar score cannot express.Shapley averages the context function over contexts, while influence evaluates it at full data.Source: §1 · p. 2, Figure 1 · p. 2
  • In an equal latent-mode instance, symmetric top-k valuation has expected excess risk exactly ρδN(k) over greedy coverage, and at least this much for non-coordinated randomized scores.Marginal within-mode symmetry alone is insufficient; LOO is zero and Shapley is mode-constant in the analyzed instance.Source: §1 · p. 2
  • When labels name the learner’s modes, balancing the same scores recovers missing coverage and can outperform label-free coverage; absent or coarser labels call for explicit geometry coverage.The practical rule is to balance within labels when they identify modes, and otherwise add a coverage correction.Source: §1 · pp. 2–3
  • Coverage helps only when target modes are expressed in the learner geometry; hidden modes and quality variation independent of coverage limit the approach.The paper also notes that a diagnostic routes between balance and coverage, while treating residual within-label structure as a caution signal.Source: §1 · p. 3

2 Value is a context function

This section defines a point’s value as a function of the selected set and gives a kernel-learner form for that value. It distinguishes when induced coverage risk equals supervised risk from when coverage is only a proxy, and explains how redundancy creates context dependence.

2 Value is a context function

  • For kernel or linear learners, selection risk is representation-coverage mass outside the selected feature span, and a point’s marginal is its feature’s Σ-novelty after orthogonalization against selected points.The residual feature direction supplies the geometry-based marginal.Source: §2 · p. 3
  • Coverage risk equals supervised excess risk under mode-aligned targets, exactly at η = 0 and to leading order for small η; for mode-constant classification labels, uncovered modes are predicted at chance.Without mode alignment, coverage should be read as a proxy rather than general supervised risk.Source: §2 · pp. 3–4
  • Redundancy makes marginal value context-dependent through rank collapse, which can reduce a point’s marginal to zero, and directional rotation, which changes its residual direction.Directional interaction also requires anisotropic Σ; exact-duplicate or same-mode rank collapse can occur even with isotropic Σ.Source: §2 · pp. 4–5
  • The interaction disappears when feature vectors are mutually orthogonal; otherwise shared learner-geometry directions carry the selection interaction that per-point diagonals omit.The paper relates this distinction to marginal screening versus forward selection under collinearity.Source: §2 · pp. 4–5
  • Modes are groups the learner must cover, while classes are labels available at selection and clusters are geometric groups; the correction depends on whether these groups coincide.Balancing labels suffices when modes are classes; finer or unlabeled geometric groups call for coverage.Source: §2 · p. 5

3 The interaction gap

In equal latent-mode models, fixed mode-blind top-k rankings under-cover relative to adaptive coverage, with an exact excess-risk gap. The gap depends on redundancy, budget, and task difficulty, while extensions and scope qualifications delimit when the result applies.

3.1 The hard latent-mode model and the exact gap

  • In equal modes, symmetric fixed top-k incurs exact excess risk ρδN(k) versus greedy coverage.Greedy covers min(k, r) modes in this instance.Source: §3 · p. 5, §3.1 · p. 6
  • In redundant equal modes, LOO is zero and Shapley is ρ/m, while a point’s true marginal switches between ρ and zero with context.Block exchangeability makes symmetric valuations mode-constant.Source: §3.1 · pp. 5–6
  • The gap is positive exactly for a hard task, redundant modes, and budgets 2 ≤ k ≤ N − m; it is largest when k = Θ(r).It vanishes outside that range and is a constant fraction of total risk at the binding budget.Source: §3.1 · p. 7

3.2 Extensions: general randomized scores and soft modes

  • For equal modes, non-coordinated randomized scores retain the lower bound, while soft-mode perturbations preserve a positive gap for sufficiently small η.Coordinated scores can beat the occupancy bound by selecting one representative per mode, but this requires mode-aware structure.Source: §3.2 · p. 7

3.3 Scope and qualifications

  • The critique is limited to fixed, mode-blind rankings; unequal mode sizes can reverse the bound, and partial alignment remains an open case.Adaptive recomputation and mode-aware grouping escape the fixed-ranking regime.Source: §3.3 · p. 8

4 Experiments

Experiments focus on the binding-budget regime, using frozen pretrained representations and probe-family learners, with additional CNN and LoRA checks and real-data evaluations on frozen ImageNet ViT-B/16 features. Across settings, they examine when balancing suffices, when explicit coverage is needed or fails, and find that coverage predicts accuracy while the coverage gap is nearly unchanged between ridge and MLP learners.

4.1 Direct validation of the theory

  • In the synthetic latent-mode setting, coverage spans 9.7/10 classes versus 6.0 for Shapley, and the gap peaks at k = C before collapsing as redundancy falls or the budget loosens.The gap appears under a binding budget, hard task, and redundant geometry, reaching +0.278; as effective rank rises from 15.5 to 29.8, it collapses from +0.35 to within 0.025 of zero.Source: §4.1 · pp. 9–10, Figure 2 · p. 10

4.2 The modes-vs-labels boundary (real ViT-B/16 features)

  • When labels name modes, class-balancing the same scores matches or beats greedy coverage; without labels, coverage beats random by +0.08 to +0.20 while raw top-k falls below random.With coarse labels, coarse-balanced facility location reaches 0.419 fine accuracy, above label-free coverage at 0.392 and below only the unavailable fine-label oracle at 0.474.Source: Table 3 · p. 11, §4.2 · pp. 11–12, Table 5 · p. 12

4.3 Failure modes: when neither correction works

  • Coverage fails when learner representations do not express the relevant modes or when data quality varies independently of coverage.On Waterbirds, coverage methods have worst-group accuracy 0.21–0.23 at k = 20, below random at 0.36 and label-balancing at 0.43; on corrupted pools, quality-then-coverage avoids outliers and performs best.Source: §4.3 · p. 13, Table 7 · p. 13

4.4 The diagnostic: balance or cover, decided before selecting

  • A pre-fixed alignment diagnostic routes between balancing and coverage without test accuracy, while near-threshold coarse-label cases trigger reporting both safe selectors.The coarse-label case misroutes at a = 0.466 against τ = 0.45; its wLMM rise from 10.3 to 14.2 is the stated ambiguity signal, and the Waterbirds subgroup audit detects hidden modes when subgroup annotations exist.Source: §4.4 · pp. 14–15, Table 8 · p. 15

4.5 Broader evidence and robustness

  • Coverage’s benefits persist across learner and encoder variants, and classes covered positively rank-correlate with accuracy across 244 pooled pairs (Spearman +0.60 to +0.94; mean +0.80).The gap also appears with an SGD-trained MLP, logistic regression, k-NN, multiple encoders, text, a from-scratch CNN, and a LoRA fine-tune.Source: §4.5 · p. 15
  • The coverage gap (greedy-FL − random) is nearly identical under ridge and an SGD-trained MLP, while raw leverage top-k stays below random under both.Label-free selection is evaluated at k = C; MLP results use 5 seeds with ± se.Source: Table 9 · p. 16
  • +0.80 mean per-setting Spearman r links class coverage with test accuracy across 244 pairs pooled from six settings.Raw scalar top-k concentrates at low coverage and accuracy, while balanced and coverage selectors restore both.Source: Figure 4 · p. 16
  • Real-data results use frozen ImageNet ViT-B/16 features over 3–5 seeds, and selectors are few-line numpy routines.Appendix F provides full hyperparameters and pool/budget construction; code and exact configurations are available at the cited repository.Source: §4.5 · p. 16

5 Related work

The paper distinguishes fixed per-point rankings from adaptive set-level coverage, locating when class balance can serve as a proxy for coverage. It connects this boundary to data valuation, coreset selection, information-theoretic selection, and data pruning.

Per-point data valuation

  • The paper’s contribution is an exact occupancy gap and a modes-vs-labels boundary for deciding when class balance suffices and when explicit coverage is needed.It limits the claim to fixed, mode-blind, non-adaptive top-k rankings in the redundant latent-mode regime.Source: §5 · p. 17

Coresets and data pruning

  • Set-level coverage addresses the selection problem by choosing a subset, while difficulty-based pruning scores remain per-point rankings.The paper supplies a theory for when coverage is needed and when class balance is an adequate proxy.Source: §5 · p. 17

Information-theoretic selection

  • Adaptive information-theoretic selection is set-level coverage, whereas using a fixed per-point ranking inherits the occupancy gap.Greedy EPIG matches facility location on CIFAR-100 (0.411 vs. 0.403), while fixed top-k EPIG falls below random (0.216 vs. 0.280).Source: §5 · p. 17

Data selection for LMs; marginal screening

  • The boundary predicts when balancing groups is enough and when representation-based coverage is needed for language-model data selection.The paper relates the gap to univariate screening under collinearity and identifies its exact occupancy form.Source: §5 · p. 17

Subgroup robustness

  • Waterbirds illustrates the boundary when a generic representation does not express subgroup modes.The relevant subgroups are not expressed by the representation, matching the paper’s learner-geometry condition.Source: §5 · p. 17

6 Limitations and conclusion

The paper scopes its conclusions to primarily frozen-representation learners and cautions that coverage depends on what the learner geometry expresses and on data quality. It closes with a practical balance-or-cover rule and notes representation and subgroup auditing as broader concerns.

6 Limitations and conclusion

  • The conclusions primarily concern non-adaptive valuations and frozen-representation learners, with additional pilots extending selector-ordering observations to from-scratch training and parameter-efficient fine-tuning.Full-scale fine-tuning at larger budgets and LLM validation remain natural next steps.Source: §6 · p. 18
  • When point quality varies independently of coverage, pure coverage may select unique but low-quality outliers, motivating quality filtering before coverage.Formalizing the quality–coverage trade-off is left as a natural extension.Source: §6 · p. 18

Broader Impact Statement

  • Geometric coverage inherits representation blind spots, so subgroup representation should be audited rather than assuming diversity objectives protect minorities.The paper also notes that the balance-or-cover rule may be easier to inspect and contest than opaque scores.Source: Broader Impact Statement · p. 18

Appendix

  • The appendix supplies proofs for the main results and extensions, experimental details and robustness checks, plus an information-theoretic interpretation and tests of the mode-aligned-target assumption.It includes the closed form, latent-mode and interaction-gap results, and a soft-mode extension.Source: Appendix · p. 20

A The coverage risk and the marginal-value closed form

  • For kernel or linear learners, selected points must reconstruct the feature second moment, and risk is the uncovered second-moment mass.The marginal value is the Σ-Rayleigh quotient of a point’s feature after orthogonalization against the selected set; redundancy makes it context-dependent through residual rotation or rank collapse.Source: A The coverage risk and the… · pp. 20–21

B The latent-mode instance: Lemmas 1–2

The latent-mode instance makes marginal value depend on whether a mode is already covered. Symmetric valuations collapse to mode-constant scores, so scalar rankings cannot represent the context-dependent marginal.

B The latent-mode instance: Lemmas 1–2

  • In the block learner geometry, covering one point eliminates a mode’s residual, while each uncovered mode incurs a positive risk cost.The risk reduces to the sum of costs for uncovered modes.Source: B The latent-mode instance: Lemmas 1–2 · p. 21
  • For a point in mode j, its marginal is ρ_j if that mode is uncovered and zero otherwise, so its interaction is ρ_j.A fixed scalar disagrees with the true marginal on an Ω(1) fraction of contexts.Source: B The latent-mode instance: Lemmas 1–2 · pp. 21–22
  • Any valuation that is a symmetric functional of the training distribution assigns the same score to every point within a mode.Within-mode exchangeability makes the points distributionally identical under relabeling.Source: B The latent-mode instance: Lemmas 1–2 · p. 22

C The two canonical scalars, computed exactly

The appendix computes leave-one-out and Data Shapley valuations in the latent-mode setting. Leave-one-out assigns zero to every point when each mode has redundant examples, while Shapley averages the mode’s first-arrival contribution.

C The two canonical scalars, computed exactly

  • When each mode has at least two points, leave-one-out assigns zero to every point because removing one leaves its mode covered.In this redundant regime, top-k is then an arbitrary tie-break.Source: C The two canonical scalars, computed… · p. 22
  • Data Shapley gives a point in mode j value ρ_j times the probability it is first from that mode in a random permutation.For equal modes, the value is constant across all points; for unequal modes, it remains mode-constant and up-weights rare modes.Source: C The two canonical scalars, computed… · p. 22

D Proof of the interaction gap

In the equal-mode model, greedy coverage selects a fresh mode while one remains, whereas fixed mode-constant rankings cover no more modes than uniform random selection. Their risk gap is positive only in a specific budget, redundancy, and task regime, and its magnitude varies with the budget.

D Proof of the interaction gap

  • A symmetric valuation incurs exactly ρδN(k) excess risk over greedy coverage, while non-coordinated scores incur at least that gap for equal-sized modes.Greedy coverage covers min(k, r) modes; the positivity range is ρ > 0, m ≥ 2, and 2 ≤ k ≤ N − m.Source: D Proof of the interaction gap · pp. 22–23
  • A score that spreads budget across rare modes can improve coverage over uniform selection, while one that concentrates budget in a few modes can reduce it.The unconditional lower bound is specific to equal-sized modes; with unequal sizes, whether mode signal helps depends on how it allocates budget.Source: D Proof of the interaction gap · p. 23
  • For equal modes, the gap is zero at k = 1 and when k > N − m; it is positive for 2 ≤ k ≤ N − m when m ≥ 2 and ρ > 0.If m = 1, every point is its own mode and the gap is always zero.Source: D Proof of the interaction gap · pp. 22–23
  • At small budgets the per-budget regret is at most (k − 1)/(2r), while at k = αr it is Θ(r) and for k ≥ r it decays exponentially in k/r.At α = 1, δ(r) = r/e + O(1).Source: D Proof of the interaction gap · p. 23
  • The gap is a constant fraction of excess risk when the budget binds, the task has positive mode cost, and the learner geometry is redundant.The stated conditions are k = Θ(r), ρ > 0, and m ≥ 2.Source: D Proof of the interaction gap · p. 24

E Soft modes: the spiked-covariance extension (asymptotic)

The spiked-covariance extension treats modes as orthogonal directions with bounded orthogonal noise and analyzes the small-noise regime. It gives a leading-order interaction gap with an explicit residual bound, while relating the gap to the spectrum’s effective rank.

E Soft modes: the spiked-covariance extension (asymptotic)

  • With equal mode masses and assumptions of orthogonal spikes and bounded orthogonal noise, an uncovered point has marginal value at least π_j − cη², while a covered point has value at most cη².Thus, to leading order, the marginal is π_j when the mode is uncovered and O(η²) when covered.Source: E Soft modes: the spiked-covariance… · p. 24
  • The perturbation proof tracks only leading-order behavior; its O(η²) constants are bounded uniformly over selected contexts but are not sharply computed.The constants depend on boundedness assumptions and the number of selected points.Source: E Soft modes: the spiked-covariance… · p. 24
  • For η ≤ η0 and 2 ≤ k ≤ N − m, the soft-mode excess-risk bound is ρδN(k) − cη²k and is positive when η² < ρδN(k)/(ck).The paper presents this as an asymptotic extension, not a sharp finite-η result.Source: E Soft modes: the spiked-covariance… · p. 24
  • The gap is large for a peaked spectrum with low effective rank and vanishes as the spectrum flattens toward near-orthogonal points.For general covariance, the passage relates the problem to marginal screening versus orthogonal matching pursuit, governed by Gram coherence.Source: E Soft modes: the spiked-covariance… · p. 25

F Experimental details

The experiments use frozen pretrained features and compare fixed per-point rankings, balanced rankings, and adaptive coverage, with additional checks of the theory’s selector correspondence. On modes-equal-label tasks, class-balanced prototypicality achieves the highest mean accuracy on every reported dataset; the theory’s literal projection selector can diverge from facility location on real features.

F Experimental details

  • Exact KNN-Shapley scores are computed once per seed, then compared as raw top-k versus class-balanced selection using the same values.The valuation set is a held-out validation split disjoint from both the selected subset and final test set.Source: F Experimental details · p. 25
  • Each seed uses a fresh pool of at most 5,000 points, frozen ViT-B/16 embeddings projected to 64 dimensions, and budgets k ∈ {C, 2C, 4C}.The datasets include CIFAR-10/100, Oxford Pets, Flowers-102, Food-101, and Waterbirds; results average 3–5 seeds.Source: F Experimental details · p. 25
  • In the hard latent-mode instance, greedy facility location and greedy projection both select fresh modes and achieve identical risk at every budget.The empirical relationship is checked in the near-block synthetic regime and diverges on real features where the soft-mode condition fails.Source: F Experimental details · p. 25
  • Facility location uses an N × N Gram matrix, while the implemented greedy projection has cost O(Nkd + Nd²); standard graph and greedy approximations are noted for larger pools.The passage identifies facility location’s O(N²) time and memory as the relevant scaling cost.Source: F Experimental details · p. 26
  • On real features, greedy projection underperforms facility location and behaves like k-center, as trace-reduction favors spectral extremes rather than label-carrying density modes.The cited passage states that the correspondence degrades as the lemma’s soft-mode condition fails.Source: F Experimental details · p. 25, Table 10 · p. 26
  • Class-balanced prototypicality has the highest mean accuracy on every dataset; raw scalar scores lose to balanced versions and often to random, while coverage is matched or beaten when modes equal labels.Data Shapley is omitted on the 100-class datasets for cost.Source: Table 11 · p. 27

G Additional robustness experiments

Additional experiments test whether the coverage and balancing patterns persist across score families, learners, budgets, representations, and training setups. The reported patterns generally persist, while accuracy gains from balancing also depend on the quality of the within-class ranking.

G Additional robustness experiments

  • Across 18 training-dynamics score and orientation cases, raw top-k concentrates below random, while class-balancing restores full class coverage and improves accuracy in 17 cases.Balancing yields good accuracy only when the inherited within-class ranking is quality-sane; EL2N-easy balanced is best in its family on all three datasets.Source: G Additional robustness experiments · p. 27, Table 12 · p. 28
  • The label-free coverage gap persists across ridge, logistic regression, and 1-NN, while raw leverage top-k remains below random for each learner.Reported coverage gaps range from +0.12 to +0.14 on CIFAR-100 and +0.20 to +0.23 on Flowers-102.Source: G Additional robustness experiments · pp. 27–28, Table 13 · p. 28
  • Across budgets, label-free coverage beats random on every dataset with shrinking margins, while best-balanced selection matches or beats facility location at every budget.For CIFAR-100, the label-free margin is +0.131 → +0.098 → +0.061 across k = C, 2C, 4C.Source: G Additional robustness experiments · p. 28, Table 14 · p. 29
  • The coverage gap remains essentially flat across a fourfold bandwidth range and a doubling of PCA dimension, and remains positive at d = 32.The reported d = 32 gaps are +0.062 and +0.129.Source: G Additional robustness experiments · p. 28, Table 15 · p. 29
  • Random selection’s observed coverage matches the occupancy prediction on real pools, and every raw scalar top-k falls below that prediction.The table reports a maximum discrepancy of 0.24 classes and a gap of up to 49 classes for prototypicality on Flowers.Source: Table 17 · pp. 29–30
  • On 20 Newsgroups, raw scalar top-k falls below random, greedy coverage beats random, and balanced prototypicality is best.At k = C = 20, coverage is 0.358 for greedy coverage versus 0.275 for random; balanced prototypicality reaches 0.426.Source: G Additional robustness experiments · pp. 29–30, Table 18 · p. 30
  • The selector ordering persists in a from-scratch CNN pilot and in LoRA fine-tuning, with leverage below random and balanced prototypicality strongest in the reported comparisons.At k = C in the LoRA experiment, leverage top-k is 0.201, random 0.253, greedy-FL 0.380, and balanced prototypicality 0.488.Source: G Additional robustness experiments · pp. 30–31, Table 20 · p. 31, Table 19 · p. 31
  • Across three backbones, greedy coverage beats random and balanced prototypicality is best in every cell; raw leverage falls below random in five of six cells.On CLIP/CIFAR-100, leverage instead ties random, consistent with representation-dependent severity.Source: G Additional robustness experiments · p. 31, Table 21 · p. 31

H Numerical verification

The synthetic suite reproduces the predicted failures of fixed per-point selection and the gains from adaptive coverage, while direct occupancy tests and boundary checks match the theory.

H Numerical verification

  • Across random contexts, marginal value has spread/|mean| ≈ 1.5 and is bimodal, while leave-one-out is zero and Data Shapley equals ρ/m to three digits.Soft-mode marginals are 0.117 for uncovered modes and 0.004 = O(η^2) for covered modes.Source: H Numerical verification · p. 32
  • At k = C, greedy coverage scores 0.895 versus 0.637 for best per-point top-k, a +0.258 gap; Data Shapley top-k dips below random near k = C.The gap collapses from +0.35 to within 0.025 as effective rank rises from 15.5 to 29.8.Source: H Numerical verification · p. 32
  • Across the tested range, the excess-risk gap is positive for 2 ≤ k ≤ N − m, but vanishes for k = 1, k > N − m, or m = 1.For k ≥ r, the loose-budget decay is δ(k) = r(1 − 1/r)^k; at r = 10, δ(r) = 3.49 versus r/e = 3.68.Source: H Numerical verification · p. 32
  • Uniform k-subsets with r = 10 equal modes of size m = 5 match the closed-form expected coverage within 0.007 modes across k.Greedy covers min(k, r); with unequal mode sizes, a rare-favoring score covers 9.0 modes versus 1.8 for uniform sampling.Source: Table 22 · p. 32, H Numerical verification · p. 32

I.1 Coverage as Bayesian experimental design

The section interprets coverage as Bayesian experimental design: integrated posterior predictive variance is the coverage objective, and its context function is expected variance reduction. Adaptive, target-aware EPIG matches facility location, whereas fixed EPIG ranking falls below random.

I.1 Coverage as Bayesian experimental design

  • In the noiseless Gaussian linear model, coverage equals integrated posterior predictive variance, the V-optimality criterion of Bayesian design.The context function is expected variance reduction from adding a candidate point.Source: §I.1 · p. 32
  • EPIG replaces target-input variance reduction with mutual information between candidate and target labels; target-agnostic information gain instead scores a candidate by posterior variance without target-distribution weighting.The target-agnostic objective lands between random and facility location, and at random on CIFAR-100.Source: §I.1 · p. 33
  • Because the Gaussian posterior covariance is label-independent, all three information-theoretic objectives can be run label-free.With σ² = 1, greedy EPIG weakens toward its fixed ranking but remains above random.Source: §I.1 · p. 33
  • With σ² = 0.1, adaptive, target-aware greedy EPIG matches facility location on all three datasets, while fixed EPIG ranking falls below random.The fixed ranking covers fewer classes than random on every dataset.Source: Table 23 · p. 33, §I.1 · p. 33

I.2 A direct test of the mode-aligned-target assumption

The direct tests find substantial but imperfect alignment between learner targets and geometry modes, with risk and classification outcomes reflecting that distinction. The squared-loss relationship is attenuated in real geometry, while uncovered classes remain near chance under argmax readout.

I.2 A direct test of the mode-aligned-target assumption

  • The test evaluates the mode-aligned-target assumption and both Proposition 1 claims on five modes-equals-labels datasets using three seeds.The proposition predicts uncovered mode mass equals squared-loss excess risk and that an uncovered mode is predicted at chance under argmax readout.Source: §I.2 · p. 33
  • Across real features, 46%–74% of target variance is constant within label-free modes, and alignment orders the gains of facility location over random across three datasets.The gains are +0.20 on Flowers-102, +0.13 on CIFAR-100, and +0.08 on Food-101; coarse CIFAR-100 targets align more with fine-level modes (A = 0.63) than coarse clusters (A = 0.45).Source: Table 24 · p. 33, §I.2 · p. 34
  • At k = C, in-span excess risk is linearly related to uncovered class mass (R² = 0.73–0.98), but slopes of 0.03–0.28 are below the hard-mode value of 1.The attenuation is strongest where alignment is weakest, consistent with uncovered classes being partly predictable from covered neighbors.Source: Table 24 · p. 33, §I.2 · p. 34
  • Under argmax readout, the ridge probe is correct on only 0–5% of test points from uncovered classes, at or below chance.The passage identifies this as the mechanism behind the coverage–accuracy relation.Source: §I.2 · p. 34

Related papers

Loading 2610.34801v1…