PaperlayerView exact revision on arXiv

Source-linked AI summary

Predicting What You Already Know Helps: Provable Self-Supervised Learning

Jason D. Lee, Qi Lei, Nikunj Saunshi, Jiacheng Zhuo

arXiv:2008.01064v2cs.LGstat.ML

TL;DR

Self-supervised learning uses input-derived pretext tasks, but why predicting already-known information helps downstream prediction remains unclear. This paper analyzes reconstruction-based tasks under approximate conditional independence and proves guarantees for representations that support downstream prediction with a linear layer.

  • Problem

    Why predicting input-derived pretext information helps downstream learning remains unclear, despite its use in self-supervised learning.

    Source: §1 · p. 2

  • Method

    The paper analyzes reconstruction-based pretext prediction under approximate conditional independence, where predicting X2 from X1 implicitly encodes Y and Z.

    Source: §1 · p. 2

  • Results

    Under approximate conditional independence, the authors prove small representation error, allowing the learned representation to almost linearly separate the downstream task.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Approximate conditional independence is a sufficient condition for a useful pretext task, while investigating other mechanisms remains future work.

    Source: §8 · p. 18

    The full-column-rank condition requires d2≥km, which bounds the allowable size of Z.

    Source: §4 · p. 9

Abstract

from arXiv · show

Self-supervised representation learning solves auxiliary prediction tasks (known as pretext tasks) without requiring labeled data to learn useful semantic representations. These pretext tasks are created solely using the input features, such as predicting a missing image patch, recovering the color channels of an image from context, or predicting missing words in text; yet predicting this \textit{known} information helps in learning representations effective for downstream prediction tasks. We posit a mechanism exploiting the statistical connections between certain {\em reconstruction-based} pretext tasks that guarantee to learn a good representation. Formally, we quantify how the approximate independence between the components of the pretext task (conditional on the label and latent variables) allows us to learn representations that can solve the downstream task by just training a linear layer on top of the learned representation. We prove the linear layer yields small approximation error even for complex ground truth function class and will drastically reduce labeled sample complexity. Next, we show a simple modification of our method leads to nonlinear CCA, analogous to the popular SimSiam algorithm, and show similar guarantees for nonlinear CCA.

1 Introduction

Self-supervised learning uses input-derived pretext tasks, but why predicting known information helps is unclear. The paper argues approximate conditional independence can explain this benefit and yield representations that support downstream prediction with a linear layer and few labels.

1 Introduction

  • Predicting X2 from X1 can implicitly encode Y and latent variables Z, then use them to predict X2 when X1 and X2 are approximately conditionally independent given Y, Z.The introduction illustrates this with image colorization and building inpainting.Source: §1 · p. 2
  • Under approximate conditional independence, the paper proves small representation error and low labeled-sample requirements for downstream prediction using a learned representation.The analysis targets expressivity and sample complexity, with a linear predictor on top of the representation.Source: §1 · p. 2
  • For topic modeling, an appropriate approximate conditional-independence assumption yields downstream sample complexity O(k), even with a potentially infinite mixture of k topics.The two views are chosen as halves of a text document.Source: §1 · p. 3
  • The paper extends its analysis to a SimSiam variant framed as nonlinear CCA and relates the problem to decomposing the conditional distribution X2 | X1.It also connects this approach to the alternating conditional expectation algorithm.Source: §1 · p. 3
  • The paper quantifies approximate conditional independence with a partial covariance matrix, reports a risk bound linear in it, and describes simulations, text experiments, and image comparisons.The introduction says the pretext task helps when conditional independence is approximately enforced in text, and the image-based linear model performs at least as well as many baselines.Source: §1 · p. 3

1.1 Related work

  • Unlike prior contrastive-learning guarantees, this work analyzes a reconstruction objective and downstream risk under approximate conditional independence [AKK+19] [TKH20b].The overview says the main analysis also considers latent variables, finite pretext and downstream samples, several function classes, and both regression and classification.Source: §1.1 · pp. 3–4, §1.2 · p. 4

2 Preliminary

The section establishes notation and formalizes self-supervised learning as first learning a representation from pretext prediction, then fitting a linear predictor for the downstream label.

2.1 Notation

  • The notation distinguishes scalars, vectors, matrices, and random variables, and defines PX, L2(P), asymptotic notation, and the vector or matrix norm.The norm is the ℓ2 norm for vectors and Frobenius norm for matrices.Source: §2.1 · p. 5
  • EL[Y|X] denotes the best linear predictor, whereas E[Y|X] denotes the best predictor without the linearity restriction.The latter minimizes expected squared prediction error over functions of X.Source: §2.1 · p. 5
  • Partial covariance between X and Y given Z captures their correlation while setting aside Z’s effect.Source: §2.1 · p. 5

2.2 Setup and methodology

  • In the setup, X1 is the input, X2 the pretext target, and Y the downstream label; the method learns a representation for predicting X2, then regresses Y linearly on it.The pretext representation targets the best predictor within a chosen function class H.Source: §2.2 · p. 5
  • Downstream performance separates approximation error, measuring linear predictability of the optimal label predictor from ψ, from estimation error measured by excess risk on n2 labeled samples.The samples are drawn independently from PX1Y.Source: §2.2 · p. 6

3 Guaranteed recovery with conditional independence

Under conditional independence of the input and pretext target given the label, reconstruction can yield representations that support downstream prediction with a linear layer. The section establishes this first for idealized settings, then examines universal and feature-map-restricted function classes.

3 Guaranteed recovery with conditional independence

  • The section uses conditional independence as an idealized, strong assumption to clarify its role, then analyzes the general case with universal or feature-map-restricted functions.Its general recipe is to characterize the optimal pretext predictor, use conditional independence to establish downstream linearity, and exploit low rank for estimation guarantees.Source: §3 · p. 6

3.1 Universal function class.

  • Under X1 ⊥ X2 | Y and rank-k class-conditional means of X2, the optimal conditional-mean representation ψ* gives zero approximation error, making even nonlinear f* linear in ψ*.The representation implicitly encodes the label probabilities given X1; the result assumes ψ* is learned perfectly from abundant unlabeled data.Source: §3.1 · p. 7

3.2 Function class induced by feature maps.

  • Under Assumption 3.1 and rank condition k=|Y|, conditional independence (CI) preserves ψ*’s approximation error at φ1’s level, while SSL reduces the labeled-data requirement from complexity for D1 to O(k).Theorem 3.5 states that n2 ≫ ρ^4(k + log(1/δ)) under the same assumption as Lemma 3.4, Assumption 3.2 for ψ*, and Assumption 3.3.Source: §3.2 · p. 8

4 Beyond conditional independence

The section generalizes exact conditional independence to approximate conditional independence, allowing extra latent variables while requiring the pretext target to correlate with them. Under small dependence and pretext errors, the resulting representation supports downstream prediction with excess-risk guarantees and reduced labeled sample complexity.

Beyond conditional independence

  • When approximate conditional independence and pretext-task errors are small, the section states that n2 = O(d2) labeled samples suffice for small downstream error.Without SSL, learning a complicated representation from scratch requires many more labeled samples.Source: §4 · p. 9

Approximate conditional independence

  • Approximate conditional independence permits a latent variable Z alongside Y, but requires X2 to correlate with [Y, Z], restricting how large Z can be.The full-column-rank condition requires d2 ≥ km.Source: §4 · p. 9
  • The section defines approximate conditional independence using a feature-map-weighted conditional covariance for H1 and a conditional-mean discrepancy for Hu.Exact conditional independence makes the defined error zero in both cases.Source: §4 · p. 10

Bound on excess risk

  • n1,n2 ≫ ρ^4(d2 + log(1/δ)) is the sample-size condition under which Theorem 4.2 bounds downstream excess risk when pretext representation error is controlled.The risk includes labeled-sample estimation error and approximation error from imperfect conditional independence and imperfect pretext learning.Source: §4 · p. 10
  • Keeping only the top km principal components of the learned representation further improves the excess-risk bound.The high-signal features derive from Y and Z, while approximate conditional independence and pretext error contribute approximation error.Source: §4 · p. 10
  • SSL reduces labeled sample complexity from the complexity of the downstream function class H to Õ(km), avoiding the cost of learning a complicated representation from scratch.The approximation-error terms may require fine-tuning to remove when enough downstream labeled data are available.Source: §4 · pp. 10–11

5 Example: Topic Modeling

This section instantiates reconstruction-based self-supervised learning for topic models. Its theoretical bound requires only O(k) labeled samples for the downstream task.

5 Example: Topic Modeling

  • A document’s topic mixture is sampled from τ, then each word is generated by sampling a topic from that mixture and a word from the topic’s vocabulary distribution.Each of the k topics is represented by a distribution over the vocabulary.Source: §5 · p. 11
  • The reconstruction task predicts the second half’s normalized bag of words from a linear transformation of the first half’s representation.The document is split evenly, and the representation ψ(X1) is trained to minimize squared error against X2.Source: §5 · p. 11
  • The downstream target is a linear function of the document’s expected topic mixture, with zero-mean σ²-subgaussian noise.The target is Y = wᵀE[µ|X] + N.Source: §5 · p. 11
  • Words are sampled independently given topic mixture µ, so the two document halves are conditionally independent given µ.The corollary further gives a latent variable under which the halves are uncorrelated and E[Y|X1] is linear in the latent posterior.Source: §5 · pp. 11–12
  • O(k) samples suffice for the downstream task under the topic-model instantiation of the framework.The section derives bounds for the framework’s ϵCI and β quantities from the topic-model generative process.Source: §5 · p. 12

6 Conditional distribution decomposition: SimSiam, CCA, ACE

This section connects reconstruction-based prediction to nonlinear CCA, SimSiam, and ACE, relating the learned representations to the conditional distribution of X2 given X1. It extends the analysis to nonlinear CCA and characterizes ACE through an operator decomposition.

6.1 Theoretical guarantees for non-linear CCA

  • Instead of fixing η, the method can jointly learn an η that is easily predicted from X1, yielding a formulation corresponding to nonlinear CCA.The setup also captures SimSiam, which uses no negative samples.Source: §6.1 · p. 12
  • For augmented-image pairs, nonlinear CCA resembles SimSiam [CH21], but uses representation normalization rather than stop-gradient to prevent collapse, so negative samples are unnecessary.CCA maximizes the representations’ inner product for positive pairs, while normalization prevents a trivial collapsed representation.Source: §6.1 · p. 13
  • Theorem 6.1 gives an approximation-error guarantee for ψ in nonlinear CCA, and Corollary 6.2 gives a generalization bound when ψ satisfies Assumption 3.2.The bound can be derived by choosing gy such that E[gy(X2)|Y = y] = 1(Y = y).Source: §6.1 · p. 13
  • Under α-Bayes error, if the joint distribution of X1 and X2 is non-degenerate, λ < 1, and small Bayes error yields good downstream performance.Here λ is the k-th maximal correlation between X1 and X2.Source: §6.1 · p. 14

6.2 Connection to ACE algorithm and maximal correlation

  • ACE [MKHZ15, BF85, Buj90] solves the (k + 1)-SVD of the conditional-expectation operator T; its top singular vectors are constant, and generalized ACE is equivalent to nonlinear CCA.The section identifies SVD of T with the alternating conditional expectation algorithm.Source: §6.2 · p. 15

7 Experiments

Experiments test whether self-supervised representations support downstream prediction when pretext-task components are approximately conditionally independent. Synthetic simulations examine risk scaling under controlled dependence, and Yearbook experiments compare inpainting-derived representations with direct image inputs.

Simulations

  • Synthetic data varied dependence between X1 and X2 using α, where α = 0 made them conditionally independent given Y and α = 1 made X2 fully dependent on X1.Labels were sampled uniformly from {1, ..., k} in a Gaussian mixture setup.Source: §7 · p. 16
  • In linear-function simulations, prediction MSE using ψ(X1) scaled linearly with label cardinality k and with ϵCI.The experiments set d1 = 50, d2 = 40, n1 = 4000, and n2 = 1000, and measured excess risk with mean squared error.Source: §7 · p. 16
  • With the universal function class, excess-risk behavior likewise varied linearly with k and ϵCI, matching the linear-class findings.The comparison used ψ* = E[Y | X1] and kernel linear regression with an RBF kernel as the baseline.Source: §7 · p. 17

Computer Vision Task

  • On Yearbook date prediction, inpainting-derived ψ outperformed direct X1 and X2 inputs with both a linear model and ResNet18 [HZRS16].ψ was learned from unlabeled data and then fixed while a linear predictor was trained with a smaller labeled set; the paper reports a ResNet18 [HZRS16] mean absolute difference of 6.89 when trained on the full training set.Source: §7 · p. 17, Figure 2 · p. 17

8 Conclusion … A.3 Technical Facts for Matrix Concentration

The paper quantifies how approximate conditional independence between pretext and downstream data can improve downstream sample complexity, with support from simulated and real CV and NLP experiments. It presents approximate conditional independence as sufficient, not necessary, and leaves other mechanisms for future work.

8 Conclusion

  • The paper theoretically links approximate conditional independence between pretext and downstream data to downstream sample-complexity benefits, supported by simulated and real CV and NLP experiments.The authors note that approximate conditional independence is sufficient, not necessary, and leave other mechanisms for future work.Source: §8 · p. 18

A.1 Relation of Inverse Covariance Matrix and Partial Correlation

  • The partial covariance of X given Y is the covariance of X and Y after subtracting their conditional expectations given Y.The passage defines it as cov(X − E[X|Y], X − E[Y|Y]) and denotes it Σ_XX·Y.Source: §A.1 · p. 25

A.2 Relation to Conditional Independence

  • If X1 and X2 are conditionally independent given Y, their partial covariance given Y is zero.The passage also identifies conditional independence with a lemma adapted from.Source: §A.2 · p. 25
  • For discrete Y, conditional expectations given Y can be expressed using its one-hot encoding, including for arbitrary functions of X.This observation is used to extend the conditional-independence argument to feature maps.Source: §A.2 · p. 25

A.3 Technical Facts for Matrix Concentration

  • The appendix gives Gaussian and sub-Gaussian covariance concentration claims for an independent rank-k matrix B, under sample-size conditions n ≫ k + log(1/δ) and n ≫ ρ^4(k + log(1/δ)), respectively.Both claims state probability at least 1 − δ; the supplied passages do not show the concentration bounds themselves.Source: §A.3 · p. 26
  • For Gaussian Z projected onto a fixed d-dimensional space, the appendix uses chi-square concentration for projected columns and a union bound to establish a high-probability projection bound.The supplied passages state probability at least 1 − δ but omit the bound's expression.Source: §A.3 · p. 26

B Warm-up: jointly Gaussian variables · C Omitted Proofs with Conditional Independence · C.1 Omitted Proof for General Random Variables

The appendix shows that under conditional-independence and rank conditions, reconstruction-based representations can linearly recover the downstream predictor with zero approximation error. It extends the Gaussian warm-up to latent-variable conditions and general random variables, and gives proof mechanisms and sample-complexity conditions for these results.

B Warm-up: jointly Gaussian variables

  • Under joint Gaussianity and X1 ⊥ X2 | Y, if ΣX2Y has rank k, the optimal downstream predictor is exactly linear in ψ*, with zero approximation error.The warm-up states this as f*(x1) = W*ψ*(x1).Source: B Warm-up: jointly Gaussian variables · p. 28
  • Full column rank of ΣX2Y means X2 captures all directions of Y and is necessary for X2 to be a reasonable pretext task.In this case ψ* selects d2 of d1 features sufficient to predict Y.Source: B Warm-up: jointly Gaussian variables · p. 28
  • Self-supervised learning reduces sample complexity from ˜O(d1) to ˜O(k); under Assumptions B.1 and B.2, when n2 ≫ k + log(1/δ), the learned target-task predictor’s excess risk is assessed with probability at least 1 − δ.Source: B Warm-up: jointly Gaussian variables · p. 28
  • If X1 ⊥ X2 | [Y, Z] and ΣX2,[Y,Z] has rank k + m, the representation still has zero approximation error, and the excess-risk result replaces k by k + m.The stated assumption requires d2 ≥ k + m and treats Z as nonredundant latent information.Source: B Warm-up: jointly Gaussian variables · p. 29

C Omitted Proofs with Conditional Independence

  • The omitted Gaussian proof uses a left inverse of ΣX2Y to recover E[Y|X1] from ψ, establishing zero approximation error when ΣX2Y has rank k.The proof derives this recovery from the expression for E[X2|X1].Source: C Omitted Proofs with Conditional… · p. 29
  • The excess-risk proof writes the target as a linear function of ψ and uses a basic inequality with the residual noise projected onto ψ.The residual is mean-zero conditional on X1, and the proof invokes its Gaussian distribution with covariance ΣYY|X1.Source: C Omitted Proofs with Conditional… · pp. 29–30

C.1 Omitted Proof for General Random Variables

  • For general random variables, the proof constructs ψ from the posterior distribution of Y given X1 and the class-conditional means of X2; rank |Y| lets a pseudoinverse recover that posterior linearly.A further linear map from the posterior recovers the target representation over the support of Y.Source: §C.1 · pp. 30–31
  • In Theorem 3.2’s proof, Lemma 3.1 yields W*ψ(X1) ≡ f*(X1); it assumes centered noise with independent rows and E[∥N∥2|X1] ≤ σ2.Pψ(X1) is a projection to dimension k.Source: §C.1 · p. 31

C.2 Omitted proof of linear model with approximation error … D.3 Omitted Proof in General Setting

These omitted proofs connect approximate conditional independence to representation quality, including for practical encoder-based reconstruction. They also develop feature-map and covariance-operator tools for analyzing approximate dependence and general settings.

C.2 Omitted proof of linear model with approximation error

  • The linear-model proof decomposes Y into a fitted linear component, an approximation residual, and mean-zero noise, then uses concentration to transfer empirical loss to excess risk.The residual is a(X1) = f*(X1) − (A*)^T X1; it need not be mean zero conditional on X1.Source: §C.2 · pp. 31, 33

C.3 Argument on Denoising Auto-encoder or Context Encoder

  • With a Lipschitz decoder inverse, reconstruction error ϵ yields pretext error ϵ_pre ≤ L√ϵ, supporting the encoder E*(X1) as a downstream representation.The argument treats predicting (D*)^-1(X2) as the pretext task; conditional independence is preserved under this deterministic transform.Source: §C.3 · pp. 33–34

D.1 Warm-up: Jointly Gaussian Variables

  • The Gaussian warm-up bounds the approximation term ∥E W̄∥F by ≲ ϵ_CI/β when n ≫ k + log(1/δ).The theorem considers approximate conditional independence given the label and latent variables.Source: §D.1 · pp. 34–35

D.2 Measuring conditional dependence with cross-covariance operator

  • Universal feature maps induce function classes dense in L2(PX), while the Hilbert-Schmidt norm of a conditional cross-covariance operator quantifies approximate conditional independence.With one-hot label features, conditional independence makes the corresponding operator zero for arbitrary feature maps.Source: §D.2 · pp. 35–36

D.3 Omitted Proof in General Setting

  • For universal feature maps, the general-setting proof represents conditional expectations using linear operators on the feature space.It invokes the feature map’s density property and derives the operator through a stationary condition.Source: §D.3 · pp. 36–37

D.4 Omitted Proof for Main Results … F.1 Introducing the Operators on the Hilbert Spaces

The appendix develops guarantees for linear downstream prediction, PCA-based regression, and a topic-modeling example, then introduces Hilbert-space operators for the paper’s later conditional-distribution results.

D.4 Omitted Proof for Main Results

  • For fixed δ ∈ (0,1), Theorem D.8 states that under Assumptions 4.1 and 3.2, no approximation error (f*(X1) ≡ Aφ1(X1)), n1,n2 ≫ ρ⁴(d2 + log(1/δ)), and its stated pretext-task condition, downstream generalization holds with probability 1 −δ.The bound does not depend on d3, only on the variance of N.Source: §D.4 · p. 38
  • The proof controls representation error through conditional-independence and pretext-estimation errors, while the approximation-error case adds a term a(Xdown1).The approximation-error proof combines Theorem 3.5 with the no-approximation-error argument.Source: §D.4 · p. 39

D.5 Principal Component Regression

  • For a rank-r matrix A = L + E, its rank-r PCA approximation Ar satisfies ∥Ar − L∥F ≤ 2∥E∥F and ∥Ar − L∥2 ≤ 2∥E∥2.The appendix uses this perturbation fact to support Remark 4.1.Source: §D.5 · pp. 40–41
  • PCR accounts for finite-sample representation error and approximate conditional independence, yielding a downstream bound with projected noise and representation error terms.The target direction need not be perfectly linear in the learned or r-PCA features; the stated bound is ∥Ψr Ŵ − V − Ē W̄∥ ≤ ∥P[Ψr,L]N∥ + ∥Ē W̄∥.Source: §D.5 · p. 41

E.1 Proof for topic modeling example

  • For topic models, defining latent Ȳ by P(Ȳ|X1) = E[μ|X1] makes the conditional-independence error zero, and the conditional label mean is linear in E[Ȳ|X1].The construction uses conditional independence of the two word bags given μ.Source: §E.1 · pp. 41–42

F.1 Introducing the Operators on the Hilbert Spaces

  • The Hilbert-space setup defines operators for conditional expectations and an operator SVD whose leading functions form a representation for approximating the conditional prediction target.Theorem F.1 formulates the SVD through a rank-k approximation objective.Source: §F.1 · p. 43
  • Corollary F.2 specializes the operator result using A† applied to one-hot label indicators when the (k − 1)-th maximal correlation between X2 and Y is nonzero.The representation’s prediction target includes P(Y = y|X1).Source: §F.1 · p. 44

F.2 Proof of Theorem F.1 · G General Results and Comparison to [TKH20b]

The proof bounds the approximation error using α and a spectral-gap term, establishing how the learned representation supports a linear readout. The following section presents more general results and connects the multi-view redundancy assumption from [TKH20b] to the paper’s assumption.

F.2 Proof of Theorem F.1

  • Every linear readout of ψ can be represented as Ψ applied to a function in L2(X2), and every Tk readout has a corresponding linear weight vector.This establishes the correspondence between linear predictors on ψ and functions passed through Tk.Source: §F.2 · p. 44
  • The top eigenvalue of T is 1, so its operator norm is 1.The proof establishes this by showing that the constant function attains unit norm under T.Source: §F.2 · pp. 45–46
  • For w1,y and w2,y as in Lemma F.3, Lemma F.5 bounds the sum over y of ∥Lw2,y − w1,y∥² by 4α.Source: §F.2 · pp. 46–47
  • For Corollary F.2, choosing g*_i makes the second error term in Theorem F.1 zero, leaving only the first term to bound.The proof uses L = B ◦ A and the corresponding conditional expectation predictor.Source: §F.2 · pp. 48–49

G General Results and Comparison to [TKH20b]

  • The authors present a more general form of their results and connect the multi-view redundancy assumption from [TKH20b] to their own assumption.The supplied passage introduces this connection but does not specify the general result or the relationship between the assumptions.Source: G General Results and Comparison to… · p. 50

G.1 General Results … J.4 From E[X2|X1, Y ] = E[X2|Y ]

This section extends representation guarantees to generalized latent-variable assumptions, examining when predictions from X1 suffice for downstream tasks and how conditional-independence arguments and alternative representations enable linear recovery of the target. It also reports that ψ predicts Y with lower MSE than X1 directly, with its MSE across different n1 scaling with 1/n2.

G.1 General Results

  • Under generalized assumptions with a discrete latent variable, the representation guarantee remains as in Theorem 4.2, with an additional error term for the mismatch between E[Y|X1] and E[E[Y|Ȳ]|X1].Unlike the main-paper assumptions, Y need not itself be discrete; Ȳ must be discrete and approximately render X1 and X2, and X1 and Y, conditionally independent.Source: §G.1 · p. 51

G.2 Comparison to [TKH20b]

  • The assumptions from [TKH20b] yield meaningful guarantees when Y is nearly deterministic given X1 and X2 and the X2-related error is small; small X1-related error further supports predicting Y from X1 alone.The paper states that these assumptions can yield stronger results, including predicting Y|X1,X2 well using a representation learned from X1.Source: §G.2 · pp. 51–52

H Showing E[Y |X1] ≈E[Y |X1, X2]

  • Low Bayes error for (X1,Y) makes E[Y|X1] close to E[Y|X1,X2], with the discrepancy bounded by 2k times the Bayes error.The theorem applies under the conditions of Theorem 4.2, including a pre-optimal learned representation; the proof bounds the discrepancy by 2k Bayes-error(PX1,Y).Source: H Showing E[Y |X1] ≈E[Y |X1, X2] · p. 53

I.1 Classification tasks

  • Under Assumption I.1 and Theorem 4.2’s setting, linear regression on one-hot encodings with learned representations yields a classifier good on linear classification with probability 1 − δ, for fixed δ ∈ (0, 1).Source: §I.1 · p. 55

J Four Different Ways to Use CI; J.1 Inverse Covariance Matrix; J.2 Closed form of Linear Conditional Expectation

  • Under conditional independence and non-degeneracy, some matrix W makes E[Y|X1] equal W E[X2|X1]; the appendix presents most results for jointly Gaussian variables and extends the arguments via feature maps.For the inverse-covariance approach, conditional independence makes the relevant block matrix block diagonal, and E[Y|X1] lies in the same direction as E[X2|X1]; the closed-form derivation refers to Claim B.1 and Lemma B.2.Source: J Four Different Ways to Use CI · p. 55, §J.1 · pp. 56–57, §J.2 · p. 57

J.3 From Law of Iterated Expectation

  • Without conditional independence, the learned representation has the form ψ(x1)=Ax1+Bf*(x1), where f*(x1)=E[Y|X1]; recovering f* requires suitable properties of A and full column rank of B.With conditional independence, A=0, and f* can be recovered if X2 carries enough information about Y for ΣX2Y|X1 to have rank equal to the dimension of Y.Source: §J.3 · p. 58

J.4 From E[X2|X1, Y ] = E[X2|Y ]

  • When E[X2|X1,Y]=E[X2|Y] and the matrix of conditional means E[X2|Y] has full row rank, a linear map of the representation recovers the conditional label probabilities and hence Y.The construction uses the pseudoinverse of the conditional-mean matrix to recover the probability vector from ψ, followed by a linear map to the label support.Source: §J.4 · pp. 58–59
  • ψ consistently outperforms using X1 directly for predicting Y by MSE.The left panel compares the two approaches.Source: Figure 3 · p. 60
  • The MSE of ψ learned with different n1 scales with 1/n2, as indicated by the analysis.This comparison appears in the right panel.Source: Figure 3 · p. 60
  • Simulations are repeated 100 times, with the mean shown as a solid line and one standard error as a shadow.Source: Figure 3 · p. 60

K More on the experiments

Additional experiments examine excess risk under strict conditional independence and compare downstream performance across yearbook and SST tasks. The results report linear scaling with downstream sample size, differences between yearbook labels, and benefits from learned representations on SST.

Simulations

  • Under strict conditional independence, ψ consistently outperforms X1 for predicting Y with a linear model, and Excessive Risk scales linearly with 1/n2.The simulation fixes α = 0, k = 2, d1 = 50, and d2 = 40, and uses MSE as the empirical Excessive Risk metric.Source: K More on the experiments · p. 60

NLP Task

  • On SST, learned ψ outperforms bag-of-words φ1 in both classification and regression, especially with few labels; exact CI performs better than CI with latent variables.The experiments simulate CI by independently sampling reviews within binary-label classes, or approximate CI by sampling within fine-grained sentiment classes.Source: Figure 5 · p. 61, K More on the experiments · pp. 61–62

Computer Vision Task

  • Yearbook MSE is higher for gender than dates, while conditioning on dates cancels more covariance spectrum; the authors conjecture gender captures less dependence, increasing ϵCI and worsening downstream performance.The reported covariance norm is ∥Σ−1/2 X1X1ΣX1X2|YD∥F = 8.15.Source: K More on the experiments · p. 61, Figure 4 · p. 61

Related papers

Loading 2008.01064v2…