PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantifying the Value of Privileged Information Using a PAC-Bayesian Approach

Vasily Bokov, Sebastian Schmitt, Vedran Dunjko, Hao Wang

arXiv:2609.12891v1cs.LG

TL;DR

LUPI uses auxiliary features available during training but unavailable at deployment, yet existing guarantees are often algorithm- and setting-specific. This paper uses PAC-Bayes endpoint posteriors to quantify PI’s potential and its realization after deployment, and reports that the training-time partition gap tracks realized gains across generative and discriminative settings. The gap is necessary but not sufficient for deployment to achieve a gain.

  • Problem

    Existing LUPI guarantees are setting-specific, leaving an algorithm-agnostic information-theoretic account of how PI reshapes learning and transfers useful knowledge missing.

    Source: §1 · p. 2

  • Method

    The paper compares never-PI and always-PI endpoint Gibbs posteriors through a PAC-Bayes log-partition gap, then analyzes whether a deployment map realizes that potential.

    Source: §1 · p. 2

  • Results

    In GMM and privileged-noise GPC experiments, the training-time partition gap tracks realized gains in generative and discriminative settings.

    Source: §1 · p. 2, §5 · p. 14

  • Takeaways & Limitations

    The partition gap quantifies PI’s potential independently of a learning algorithm, while deployment determines how much of that potential is realized.

    Source: §1 · p. 2, §3.2 · p. 6

    A wide endpoint envelope is necessary but not sufficient for a LUPI deployment gain, so potential and realization require separate certification.

    Source: §3.2 · p. 6

Abstract

from arXiv · show

In practice, various learning scenarios provide access to auxiliary features exclusively during training. Incorporating such data to enhance model performance gave rise to a paradigm known as Learning Using Privileged Information (LUPI). While this extra information is intended to improve the resulting model, establishing a generalized, cohesive understanding of how privileged information (PI) transfers useful knowledge remains a challenge. Vapnik's original theory and subsequent works offer performance guarantees in certain cases, but these results are inherently per-algorithm and rely on setting-specific proof approaches. Consequently, a more general framework explaining how and when PI transfers useful knowledge is still missing. To bridge this gap, we introduce an algorithm-agnostic, information-theoretic approach based on the PAC-Bayes framework. Rather than asking whether a particular algorithm exploits PI, we ask how much value it could offer: comparing the tightest achievable risk bound with and without PI yields its potential - an upper limit on the extractable gain. We introduce a metric that quantifies this potential directly from empirical training risk, bypassing the need for test-time data access, and validate our findings in both supervised and unsupervised settings. The results demonstrate a robust correspondence between our training-time metric and true test-time performance gains. Ultimately, this work takes a necessary step toward an information-theoretic understanding of LUPI, and quantifying the potential of privileged features before committing to a model.

1 Introduction

Privileged information (PI) is available during training but may be unavailable at deployment, and existing LUPI guarantees are largely tied to particular algorithms and settings. The paper develops an algorithm-agnostic PAC-Bayes account of PI’s potential and deployment realization, then tests whether its training-time measure tracks realized gains.

1 Introduction

  • Earlier LUPI guarantees were setting-specific and focused largely on computational or sample-complexity advantages rather than an algorithm-agnostic information-theoretic account of PI.Examples include faster optimization convergence (11, 15) and computational separations using quantum hardware (1).Source: §1 · p. 2
  • The paper’s algorithm-agnostic PAC-Bayes formulation measures PI potential by the log-partition difference between idealized endpoint Gibbs posteriors and gives a condition for deployment to realize that potential.The endpoints represent never-PI and always-PI settings.Source: §1 · p. 2
  • The theoretical analysis extends from canonical Gibbs posteriors to a fixed-feature subclass of exponential-family posteriors that includes exact Bayesian posteriors for linear, generalized-linear, and Gaussian-process models.Source: §1 · p. 2
  • In GMM and privileged-noise GPC experiments, the log-partition difference tracks realized test-time gains as PI quality varies.Source: §1 · p. 2

2 Background

The background defines LUPI as training with privileged information that is unavailable at inference, and introduces the PAC-Bayes framework used to analyze such learning rules. Within that framework, Gibbs posteriors minimize the Catoni certificate, whose empirical-risk-plus-divergence budget is represented by a log-partition function.

2.1 Learning Using Privileged Information

  • LUPI allows training-time access to privileged inputs while requiring the deployed predictor to depend only on ordinary inputs; the paper treats PI as a data property whose conditional law does not depend on the learner.Source: §2.1 · p. 3
  • LUPI methods include privileged loss prediction, teacher-to-student distillation, privileged modeling of label uncertainty, and transfer through shared representations.Source: §2.1 · p. 3

2.2 PAC-Bayes Framework

  • The PAC-Bayes setup represents a randomized learning rule as a Markov kernel from samples to posterior distributions over hypotheses.Source: §2.2 · p. 3
  • For a sample-independent prior, Catoni’s theorem gives a simultaneous risk certificate for admissible posteriors, and the Gibbs posterior uniquely minimizes its bound.Source: §2.2 · p. 4
  • The Gibbs identities make the log-partition function a computable stand-in for its empirical-risk-plus-KL budget.Source: §2.2 · p. 4

3 PAC-Bayes Analysis of Privileged Information

The analysis separates PI’s potential at idealized endpoints from the portion realized after deployment without privileged inputs. It compares PAC-Bayes certificates, develops conditions for endpoint and deployment gains, and extends the framework to fixed-feature posteriors.

3.1 Basic assumptions

  • The framework assumes data-independent priors and a sample-independent deployment map, which transports PI hypotheses into predictors on ordinary inputs.A sample-dependent conditional distribution in the marginalization map would make the pushed-forward prior sample-dependent and invalidate the Catoni-bound requirements.Source: §3.1 · p. 5

3.2 Endpoint posteriors and the Gibbs envelope

  • The never-PI Gibbs posterior is the ordinary baseline, while the always-PI posterior is an oracle that minimizes the PI certificate but cannot be deployed directly.Transporting the oracle posterior through T gives the LUPI posterior; deployment can discard some or all of the PI advantage.Source: §3.2 · pp. 5–6

3.3 Partition-function criteria for PI utility

  • ΔZ > 0 exactly characterizes when the always-PI endpoint has a tighter upper certificate than the never-PI endpoint.This pre-training criterion can be evaluated from empirical risks and prior integrals, without normalizing an endpoint posterior; it does not itself establish a positive Gibbs envelope.Source: §3.3 · pp. 6–7
  • A sufficient deployment criterion accounts for both the empirical deployment gap and the KL contraction under T; when the contraction is zero, that criterion is also necessary.These certificate comparisons do not by themselves imply a true-risk ordering; a two-sided test pairs an upper certificate with a lower certificate to establish risk separation.Source: §3.3 · pp. 7–8

3.4 From endpoint Gibbs posteriors to practical algorithms

  • The theory extends beyond canonical Gibbs posteriors by absorbing a data-independent fixed-feature term into a reshaped prior, reducing the posterior to a Gibbs measure on empirical risk.The fixed-feature class includes exact Bayesian posteriors for linear, generalized-linear, and Gaussian-process models, but the data-independence condition is essential.Source: §3.4 · p. 9

4 Empirical Validation

The experiments assess whether the partition-function gap tracks realized privileged-information gains in Bayesian GMM estimation and Gaussian-process classification. In both settings, the gap tracks gains as PI quality varies, though the log-likelihood risks used in the experiments are unbounded.

4.1 Bayesian estimation of a Gaussian mixture

  • Because the negative log-likelihood risks are unbounded, the Catoni certificates do not apply literally, so the experiments use the partition gap diagnostically.The authors leave bounded-surrogate or unbounded-loss guarantees to future work.Source: §4.1 · p. 10
  • In the GMM study, privileged cluster labels reduce component ambiguity and concentrate the posterior predictive around the generating components.The study compares Bayesian predictors with and without PI and a no-PI MAP plug-in baseline.Source: §4.1 · p. 11
  • Across cluster-overlap settings, the partition gap and test log-likelihood gain grow together (r = 0.911).Figure 2 averages each setting over 14 datasets of N = 8 points.Source: Figure 2 · p. 12

4.2 Gaussian Process Classification with Privileged Noise (GPC+)

  • For GPC+, PI encodes training-label reliability as an additional, example-specific noise term; increasing PI noise progressively destroys its alignment with noisy-boundary examples.The PI scale flattens the probit likelihood for examples with low reliability.Source: §4.2 · pp. 12–13
  • As PI noise increases, the evidence gap and test-accuracy gain decrease together and vanish when PI becomes uninformative.The same diagnostic tracks realized gains in a supervised, non-conjugate, function-space model.Source: §4.2 · p. 13, Figure 3 · p. 14

5 Conclusion

The paper presents an algorithm-agnostic PAC-Bayesian account that separates PI's potential from its realization at deployment. Its experiments show the partition gap tracking gains in generative and discriminative settings, while calibrating the gap-to-gain relationship remains open.

5 Conclusion

  • The framework separates PI's potential, measured by an endpoint partition-function comparison, from how much of that advantage survives deployment without PI.The deployment correction quantifies the realized portion; two-sided certificates give sufficient conditions for a genuine separation of true risks.Source: §5 · p. 14
  • The GMM and privileged-noise GPC experiments show the training-time partition gap tracking realized gains in generative and discriminative settings.Characterizing the relationship's slope to predict gain per unit remains future work.Source: §5 · p. 14
  • The stated Catoni certificates require bounded losses, so the log-likelihood experiments use the partition gap diagnostically.The theory also assumes data-independent priors and deployment maps; computing partition functions may be difficult in large models, and endpoint potential alone does not guarantee good deployment.Source: §5 · p. 14

A.1 Gibbs Posterior Minimization

The proof identifies the minimum of the right-hand side at the Gibbs posterior and substitutes that value to establish the claim.

A.1 Gibbs Posterior Minimization

  • The right-hand side is uniquely minimized at ρ = ρλ, with value −log Zλ; substitution into Bλ proves the claim.Source: §A.1 · p. 17

A.2 Endpoint Certificate Criterion

The endpoint certificate criterion compares Catoni bounds for the never-PI and always-PI measures. A union bound makes both bounds hold simultaneously, and their ordering reduces to a condition involving empirical-risk and KL differences.

A.2 Endpoint Certificate Criterion

  • Applying Catoni's bound to each endpoint and taking a union bound makes both risk bounds hold simultaneously with probability 1 − 2ε.Source: §A.2 · p. 17
  • The endpoint bounds are ordered exactly when ∆r > ∆KL/λ; substituting the Gibbs KL identities reduces the condition to a comparison of the endpoint partition functions.Source: §A.2 · p. 17

A.3 Deployment-Certificate Criterion

The appendices establish certificate comparisons for deployment, lower and positive-envelope bounds, true-risk separation, and a fixed-feature reduction. Together, these results specify conditions for comparing privileged and ordinary learning within the PAC-Bayes framework.

A.3 Deployment-Certificate Criterion

  • The deployment criterion compares certificate arguments after confidence terms cancel, while data processing ensures deployment does not increase KL divergence and yields κT ≥ 0.The proof also establishes the equality case for Eq. (18).Source: §A.3 · pp. 17–18

A.4 Catoni Lower Certificate

  • Applying Catoni’s upper bound to the complementary loss 1 − ℓ yields a lower certificate after rearrangement.The complementary loss remains in [0, 1].Source: §A.4 · p. 18

A.5 Positive Gibbs-Envelope Certificate

  • The positive Gibbs-envelope certificate combines valid bounds and gives a sufficient condition on a∗ + b for the envelope term to be at most 1.The derivation uses 1 − e^(−cx) ≤ cx for x ≥ 0 and c > 0.Source: §A.5 · p. 18

A.6 Deployment True-Risk Separation

  • The deployment true-risk separation result gives simultaneous inequalities for ρ_lupi and ρ_never with probability at least 1 − 2ε.Both posterior events hold simultaneously because the PAC-Bayes events hold over posteriors; a union bound combines them.Source: §A.6 · pp. 18–19

A.7 Fixed-Feature Reduction

  • The fixed-feature reduction identifies the two probability measures by showing their densities relative to π are proportional and their normalizing constants agree.The density comparison uses Eq. (26) and Eq. (27).Source: §A.7 · p. 19

B More Examples of the Deployment Map

The section illustrates how deployment maps remove privileged-only components while retaining deployable hypotheses or representations. Examples distinguish teacher–student transfer, shared-feature transfer, and direct projection of a privileged branch.

Deployment-map examples

  • After hypotheses are identified by a shared parameter or latent function, the validation-study deployment map is the identity; nontrivial maps remove or average out PI-specific degrees of freedom.The identity-map cases are the GMM and GPC+ validations.Source: B More Examples of the Deployment Map · p. 19

SVM+

  • In SVM+, deployment projects the jointly learned classifier and privileged correcting function onto the classifier, and discarding the privileged branch can strictly contract KL divergence.The privileged branch models training-example difficulty but is unavailable at inference.Source: B More Examples of the Deployment Map · p. 19

Generalized distillation

  • Generalized distillation trains a teacher using privileged examples, then trains a student on ordinary inputs using both hard labels and the teacher’s soft labels.The section treats this learning rule as a frequentist point estimator rather than a Bayesian learner.Source: B More Examples of the Deployment Map · p. 20

Modality hallucination

  • Modality-hallucination methods train an RGB-input network to reproduce depth-derived logits or feature maps, then retain that network at test time.The privileged network supplies the depth-derived targets during training.Source: B More Examples of the Deployment Map · p. 20

Transfer and marginalize (TRAM)

  • In transfer and marginalize, the privileged loss trains shared representation ϕ, while deployment retains ϕ and the ordinary prediction head and discards PI-specific parameters.This transfers knowledge through the shared representation rather than teacher soft labels.Source: B More Examples of the Deployment Map · p. 20

C Examples of Fixed-Feature Distributions

The examples characterize fixed-feature distributions for exact Bayesian posteriors and several Gaussian or kernel-based models. The required conditions include proper posteriors and, for sample-trained features, data independence or an additional data-dependence argument.

Common likelihood identity and Bayesian models

  • Every proper Bayesian posterior from an i.i.d. likelihood has the fixed-feature density identity with λ = N and μ = 0 when its marginal likelihood is finite and nonzero.The examples include Bayesian linear regression and generalized linear models, with no conjugacy required for proper Bayesian GLMs.Source: C Examples of Fixed-Feature… · p. 21

Gaussian-process regression

  • For Gaussian-process regression, the exact posterior has the fixed-feature density, and its posterior mean agrees with kernel ridge under the standard noise-to-prior variance identification.The likelihood depends on finitely many measurable evaluations of the function.Source: C Examples of Fixed-Feature… · p. 21

Gaussian-process classification

  • For Gaussian-process classification, the exact posterior satisfies the density identity but is generally non-Gaussian; Laplace and expectation-propagation approximations are not automatically exact fixed-feature posteriors.The model uses a Bernoulli likelihood under the GP prior.Source: C Examples of Fixed-Feature… · p. 22

Quadratic Gibbs law, kernel ridge, and Gaussian variational posteriors

  • The common quadratic Gibbs calculation gives a Gaussian posterior whose mean and mode minimize the regularized squared-loss objective, and kernel ridge fitted values are exactly its mean.The fixed-feature construction requires the feature map, regularizer, and prior to be fixed independently of the PAC-Bayes sample; arbitrary sample-optimized covariance is not covered unless its variation matches the empirical-risk contribution.Source: C Examples of Fixed-Feature… · p. 22

Frozen last-layer networks and NTK-linearized networks

  • A frozen last-layer network and an NTK-linearized network fit the Gaussian Gibbs construction when their features or initialization are fixed independently of the sample.A same-sample-trained feature extractor or linearization point requires sample splitting or an additional data-dependence argument; a changing Jacobian is not covered by the stated claim.Source: C Examples of Fixed-Feature… · pp. 22–23

D Gaussian Mixture Details

The Gaussian-mixture example compares posterior inference with and without privileged component assignments and evaluates deployed posterior predictions. It uses exact posterior sampling with privileged assignments, while treating the partition-function difference only as a diagnostic for the unbounded likelihood loss.

GMM setup and priors

  • The GMM evaluation targets posterior inference and the deployed posterior predictive, not prediction of latent assignments; privileged information consists of the generating assignments observed during training.Both configurations use independent conjugate priors, including a Normal-Inverse-Gamma prior for component parameters.Source: D Gaussian Mixture Details · p. 23

Posterior with privileged assignments

  • Revealing privileged component assignments turns the mixture likelihood into a product of conditional terms and factorizes the posterior into three independent parts, enabling exact ancestral sampling without MCMC.The assignment-free likelihood expands into a sum over 2^N assignments, so the never-PI posterior is sampled by Metropolis–Hastings.Source: D Gaussian Mixture Details · pp. 23–24

Deployment map

  • The two configurations share a parameter space and use the identity deployment map, so κT = 0.The PI affects the loss rather than the deployment map.Source: D Gaussian Mixture Details · p. 24

Partition-function diagnostic

  • Because the negative log-likelihood is unbounded, the Catoni certificate uses a bounded surrogate and ΔZ is used only as a diagnostic quantity.The partition functions are evaluated by Monte Carlo over the prior.Source: D Gaussian Mixture Details · p. 24

Predictive comparison

  • For small N, the posterior can be multimodal, while a MAP prediction selects one mode and can be overconfident relative to the posterior predictive.The comparison is between the posterior-averaged predictive distribution and a point-estimate MAP baseline.Source: D Gaussian Mixture Details · p. 24
Loading 2609.12891v1…