PaperlayerView exact revision on arXiv

Source-linked AI summary

The Trade-off Between Covariate Dependence and Latent Structure in Representation Learning

Małgorzata Łazęcka, Ewa Szczurek

arXiv:2608.16245v1cs.LG

TL;DR

Correlated covariates create a trade-off between latent-covariate alignment and structural constraints such as latent independence. The paper formalizes this trade-off with supervised transformations and shows ordered disentanglement regimes that enable post-hoc alignment and controllable factor analysis.

  • Problem

    Correlated covariates create a tension between strong latent-covariate alignment and structural constraints, a trade-off not previously studied systematically.

  • Method

    The paper uses supervised optimization over invertible latent-space transformations to construct disentanglement regimes with different structural constraints and covariate alignment.

  • Results

    The regimes are theoretically ordered by latent-covariate dependence, while post-hoc transformations achieve stronger one-to-one dependence than naive dimension selection and iFA enables controllability.

  • Takeaways & Limitations

    The framework provides a principled way to explore controllable trade-offs between covariate dependence and latent structural constraints in pretrained embeddings and factor analysis.

  • Takeaways & Limitations

    The exclusive one-to-one regime requires the cross-covariance matrix Σ_ZY to be invertible for feasibility.

Abstract

from arXiv · show

Disentangled representation learning seeks latent representations whose indicidual dimensions each align with a distinct covariate. Unsupervised approaches typically target latent dimension independence, yet this gives no guarantee that the resulting dimensions align with semantically meaningful covariates. Supervised approaches structure the latent space using observed covariates, but under correlated covariates they cannot simultaneously control one-to-one latent-covariate alignment and latent independence. We introduce a unified, supervised framework that couples latent dimension-covariate dependence with constraints on the latent structure. Within this framework, we show an inherent trade-off, where enforcing latent independence or exclusive one-to-one latent-covariate dependence comes at a provable cost in latent-covariate alignment. We prove that the resulting disentanglement regimes are ordered by the strength of that alignment. Each regime admits a closed-form transformation of the latent space. We apply these transformations post-hoc to realign the representations of pretrained models such as CLIP, DINOv2, and ViT, and we fold them into the inference of informed factor analysis (iFA), a probabilistic model with covariate-informed factors. On simulated and real multi-omics data, we show that both post-hoc alignment and iFA enable controllability of structured latent representations.

1 Introduction

The paper studies how correlated observed covariates create a trade-off between latent dimension–covariate alignment and structural constraints such as latent independence or exclusive dependence. It introduces a supervised framework that characterizes and controls this trade-off through invertible transformations of the latent space.

  • Motivation: Disentangled representations aim to align individual latent dimensions with distinct, semantically meaningful factors of variation.Such representations support interpretability, controllable generation, and improved generalization by isolating underlying factors.
  • Core problem: When observed covariates are correlated, one-to-one latent–covariate alignment conflicts with latent independence or exclusive dependence.Exclusive dependence means each latent dimension aligns only with its corresponding covariate.
  • Related approaches: Unsupervised latent independence does not generally ensure alignment between latent dimensions and interpretable covariates.Independence is used in classical methods such as PCA and ICA and encouraged in modern nonlinear models through factorized latent representations.
  • Proposed framework: The proposed supervised framework characterizes the trade-off between latent dimension–covariate dependence and latent structural constraints to control latent representations.It formulates optimization problems over invertible latent-space transformations whose solutions yield latent–covariate dependence.

2 Related works

Related work spans independence-based disentanglement, supervision using observed or weakly supervised covariates, and supervised linear latent-variable models. Recent studies also emphasize that correlated covariates make disentanglement fundamentally different from the independent-covariate setting.

  • Independence-based disentanglement methods: Independence-based methods encourage independent representation dimensions through classical latent-variable models, probabilistic factor models, and deep generative approaches.Examples include PCA, ICA, FA, multi-view factor models such as MOFA, and related probabilistic latent-variable models.
  • Independence-based disentanglement methods: VAE-based and non-VAE approaches promote latent independence using distributional regularization, concept whitening, mutual information, or nonlinear ICA.Examples include β-VAE, FactorVAE, β-TCVAE, concept whitening, InfoGAN, and nonlinear ICA extensions.
  • Supervision in disentanglement and latent representation models: Because latent independence alone does not guarantee identifiability, other methods use weak supervision from grouped, temporal, or paired observations and stronger supervision from directly observed covariates.Conditional latent representations, such as those in CVAE, are an example of direct covariate supervision.
  • Supervised linear latent variable models: Supervised linear latent-variable models guide representations with covariates through feature selection, covariate-dependent principal components, regression, and correlation-based approaches.Supervised PCA variants include methods focusing on individually associated features and methods estimating principal components with maximal covariate dependence.
  • Disentanglement under correlated covariates: Under correlated covariates, disentanglement differs fundamentally from the independent-covariate setting, and enforcing latent independence can prevent optimal likelihood or discard information.The cited empirical studies identify correlated-covariate disentanglement as important but relatively underexplored.

3 Proposed framework

The proposed framework unifies pairwise latent–covariate alignment with structural constraints on latent representations, exploiting invertible transformations to define canonical structured representations. It characterizes four disentanglement regimes, their closed-form transformations, and an alignment ordering, then embeds the transformation family in informed factor analysis.

  • Unified framework: The framework seeks strong one-to-one alignment between latent dimensions Z_p and corresponding covariates Y_p while imposing chosen structural constraints on the latent space.Latent representations are assumed non-identifiable under invertible linear transformations, which are used to resolve representation ambiguity.
  • Disentanglement regimes: The four regimes range from unconstrained latent dependence, through intermediate interpolation and decorrelated dimensions, to exclusive latent–covariate dependence.The independence regime enforces uncorrelated latent dimensions, while the exclusive regime requires the latent–covariate cross-covariance to be diagonal.
  • Closed-form transformations: Whitening followed by an orthogonal Procrustes alignment yields a canonical independent representation, while the unconstrained and intermediate regimes admit separable closed-form transformations.The interpolation parameter λ ∈ [0, 1] recovers the decorrelating and unconstrained solutions at its endpoints.
  • Alignment trade-off: Theorem 3.1 orders the regimes by total latent–covariate alignment: dependence increases from independence through the intermediate regime to the unconstrained regime, while exclusivity is weakest.The intermediate alignment is non-decreasing in λ, and all regime alignments coincide exactly when the whitened cross-covariance columns are mutually orthogonal.
  • Probabilistic factor analysis: The framework is embedded in informed factor analysis, where covariates shape latent factors and the transformation family T_λ reshapes them.Although factors are generally dependent through the covariates, they become conditionally independent given Y because their conditional covariance is diagonal.

4 Experimental setup

The experiments test theoretical properties of transformed latent spaces and their practical use for post-hoc alignment and informed factor analysis (iFA). They use simulations, pretrained image representations, and multi-omics data, evaluating alignment, latent structure, variance explained, and disentanglement.

  • Experimental goals: Experiments target theoretical validation on simulations and practical application as post-hoc transformations and within iFA on simulated and real-world data.These are the study’s two stated experimental goals.
  • Simulation design: Four generative scenarios model realistic covariance structures among normally distributed covariates, including mixed positive and negative correlations and autoregressive dependence.The scenarios follow the model’s generative process with T = I; the passage also gives blood-panel and consecutive-time-measurement examples.
  • Image representation testbed: Post-hoc alignment is tested on CLIP, DINOv2, and ViT representations, spanning contrastive image-text, self-supervised self-distillation, and supervised ImageNet classification.Representations are extracted from 2000 Tiny-ImageNet validation observations across 200 classes, using P = 20 dummy-coded animal-category indicators as covariates.
  • Multi-omics evaluation: Real-data experiments use breast TCGA multi-omics measurements from 150 patients, with cancer subtypes Basal, Her2, and LumA as dummy-coded covariates.The data include mRNA, miRNA, and proteomics; analyses use stratified 5-fold cross-validation and within-fold standardization.
  • Baselines and metrics: Comparisons include unsupervised PCA and MOFA, supervised unpaired SPCA variants and PLS, and iVAE, while unpaired methods are aligned by solving a linear sum assignment problem.Evaluation reports average factor-covariate correlations, squared Frobenius covariance distance, average variance explained per factor, and D, C, I, and SAP disentanglement metrics.

5 Results

Results across simulations, pretrained representations, and iFA confirm that stronger latent–covariate alignment requires weaker latent independence. The proposed transformations preserve factor interpretability, outperform naive alignment, and enable efficient exploration of this trade-off.

  • Simulation results: Simulation results show that latent-structure constraints reduce achievable latent–covariate dependence, ordering regimes from strongest alignment under T∗1 to weaker alignment under T∗0 and T∗ex.T∗0 enforces uncorrelated factors, whereas T∗ex enforces exclusive one-to-one factor–covariate dependence at a further alignment cost.
  • Pretrained representations: Post-hoc transformations estimated from pretrained representations achieve stronger one-to-one dependence than selecting each original dimension’s most correlated covariate and reproduce the theoretical ordering.T∗0 whitens representations, while T∗1 trades some independence for higher covariate alignment and remains less correlated than plain Procrustes rotation.
  • iFA inference: In iFA, increasing λ increases covariate dependence while weakening factor independence, and the gain toward T∗1 is largest when covariates are strongly inter-correlated.Across β values, correlations rise from around 0.2 to 0.6, matching the true generative values; variance explained per factor remains high.
  • Real-data results: On real data, individual factors retain high variance explained across the iFAλ family, with T∗1 explaining the most and T∗0 the least.The latent-independence versus covariate-dependence trade-off persists throughout the family.
  • Disentanglement metrics: Disentanglement and completeness improve as λ grows, whereas enforcing latent independence spreads covariate-specific information across factors; informativeness instead tracks signal strength.For T∗ex, normalised SAP is high because each factor carries one covariate’s information, while unnormalised SAP is low because overall dependence is small.
  • Baselines and efficiency: The iFAλ family traces a stronger trade-off frontier than most competing methods, while remaining slower than classical baselines but faster than SOFA.After fitting the base model, multiple Tλ transformations can be applied at negligible cost, making regime exploration cheaper than refitting.

6 Conclusions

The paper introduces a supervised framework that navigates the trade-off between latent-covariate dependence and structural constraints. It identifies four ordered disentanglement regimes with closed-form latent transformations.

  • The framework couples latent-covariate dependence with structural constraints on the latent space.
  • Four regimes—independent, intermediate, unconstrained, and exclusive—arise as solutions to optimization problems.
  • Theorem 3.1 proves that these regimes are ordered by latent-covariate dependence.
  • The resulting transformations have closed-form expressions and can be applied post-hoc.

A Proofs and derivations … A.5 Summary of the defined transformations

The proofs reformulate the population orthogonal Procrustes problem, derive closed-form regime transformations, connect strong dependence to regression, establish identifiability, and summarize alignment formulas under regularity assumptions.

  • A Proofs and derivations: The population orthogonal Procrustes objective reduces to maximizing a cross term after whitening, because the remaining squared-norm terms are constant in the orthogonal transformation.The reduction uses Cov(Z) = I after whitening and trace invariance.
  • A.2 Closed-form solution of 𝑇∗: The optimal transformation T∗ has a closed-form expression obtained from an eigendecomposition.The derivation identifies the maximizer through the resulting eigendecomposition.
  • A.3 Notes on 𝑇∗ 1 : connection to regression: The strong-dependence transformation equals multivariate least-squares regression of covariates on latent variables, with attained alignment given by the multiple correlation coefficient.This coefficient is also the maximal correlation between each covariate and any linear combination of the latent variables.
  • A.4 Identifiability: For every regime, replacing Z by any invertible equivalent representation preserves the structured optimum, so T∗•(Z) identifies a canonical latent representation.The proof uses the bijection T′ = TA^-1 between transformations of equivalent latent representations.
  • A.4 Identifiability: The general solution is obtained by solving each regime in the whitened frame and composing with the whitening transformation.This yields T∗• = T∗•(Z)T∗0.
  • A.5 Summary of the defined transformations: The transformations are summarized through total alignment M and per-dimension alignment J(T,p), with formulas and diagonal normalizers collected in Table S1.The derivations assume Σ_Z ≻ 0 and full-rank Σ_ZY; under these conditions M is symmetric and positive definite.

A.6 Proof of Theorem 3.1 · B Additional model details

Theorem 3.1 is established by summing per-dimension inequalities, characterizing equality through eigenvector or diagonal-column conditions, and proving monotonicity in λ. Additional model details define shared latent structure across modalities and simulation scenarios with varied covariate dependencies.

  • A.6 Proof of Theorem 3.1: Theorem 3.1 follows by summing per-dimension inequalities and applying Lemmas A.3 and A.4, including their equality conditions and λ-monotonicity.The proof uses per-dimension representations from Section A.5, including M = (Σ′ZΣY)1/2.
  • A.6 Proof of Theorem 3.1: For symmetric positive-definite M, the two per-dimension values are M_pp and ((M^-2)_pp)^-1/2.The claim is derived using the Cauchy-Schwarz inequality twice.
  • A.6 Proof of Theorem 3.1: Equality holds exactly when e_p is an eigenvector of M, equivalently when column p of M has no off-diagonal entries.The same eigenvector condition makes both Cauchy-Schwarz steps equalities.
  • A.6 Proof of Theorem 3.1: The alignment derivative has non-negative numerator factors on [0, 1], and it vanishes only at λ = 1 or when ||M·p||2 = (M^2)_pp.The alignment is written as u(λ)w(λ)^-1/2, with w(λ) explicitly defined from M.
  • B Additional model details: In the multimodal model, each modality has its own observed matrix, loading matrix, and parameters, while the latent representation ˜Z is shared across modalities.The joint distribution factorizes over modality-specific quantities and the shared latent representation.
  • B Additional model details: The simulation study includes four scenarios covering positive-and-negative, autoregressive, positive, and negative covariate dependencies.These scenarios represent unstructured dependencies, time/space dependences, scale effects, and dummy-coded categories.

C Additional details on the numerical experiments · C.1 Simulation scenarios

The simulations generate synthetic data from Eq. (9) with T = I across four scenarios that vary covariate-covariance structure. They vary α to control covariate-correlation strength while holding other parameters fixed across scenarios.

  • C.1 Simulation scenarios: Synthetic data are generated from the generative model in Eq. (9), with T = I.The setup is illustrated in Fig. S6.
  • C.1 Simulation scenarios: The simulations include four scenarios with different covariate-covariance structures.These scenarios are summarized in Tab. S2.
  • C.1 Simulation scenarios: The covariance structures are chosen to reflect qualitatively different but realistic dependence patterns.
  • C.1 Simulation scenarios: Scenario-specific covariance structures are summarized in Tab. S2.
  • C.1 Simulation scenarios: The simulations vary α ∈ [0, 1] to control the strength of covariate correlations by interpolating the covariance structure.
  • C.1 Simulation scenarios: Parameters held fixed across all scenarios are listed in Tab. S3.Tab. S3 is titled “Simulation parameter settings.”

C.2 Metrics

The metrics quantify latent–covariate alignment, factor concentration, information preservation, and single-feature predictability using an importance matrix computed from regressors or joint covariance. They are evaluated only on informed latent dimensions, with SAP and nSAP capturing absolute and relative predictor dominance.

  • Importance matrix: An importance matrix R records each latent dimension’s contribution to predicting each covariate, using absolute Lasso coefficients for samples and joint covariance in population metrics.All metrics are computed only for informed latent dimensions aligned with covariates.
  • Alignment metrics: Disentanglement measures whether each latent dimension specializes in a single covariate, while completeness measures whether each factor is concentrated in one latent.High disentanglement indicates latent specialization; high completeness indicates concentration of each factor in one latent.
  • Information metric: Informativeness measures whether the latent representation preserves recoverable covariate information, independently of how that information is organized.It is computed as mean out-of-sample predictive performance; high informativeness indicates jointly preserved covariate information.
  • Predictability metrics: SAP averages, across factors, the gap between the highest and second-highest univariate predictive scores, measuring whether one latent dominates each factor.Unlike disentanglement and completeness, SAP uses single-feature predictability and is more sensitive to correlations among ground-truth factors.
  • Predictability metrics: nSAP isolates the relative dominance of the best predictor, distinguishing weak overall prediction from genuinely competing predictors.SAP reports an absolute gap that combines dominant-predictor strength with its lead over the runner-up.

C.3 Methods details · C.4 Image representations

Methods are compared under matched factor counts and optimization budgets, while iFA uses pretraining, early stopping, and λ-dependent fine-tuning. Image representations come from ViT, CLIP, and DINOv2 pretrained models, each providing 768-dimensional features.

  • C.3 Methods details: All comparison methods use K = 10 and 2000 iterations for simulations, and K = 20 and 5000 iterations for real data.The matched budgets are applied across methods to keep comparisons fair.
  • C.3 Methods details: Methods without covariate-paired factors have their informed factors selected by H.These methods include PCA, MOFA, PLS, both supervised PCA variants, and iVAE.
  • C.3 Methods details: iFA uses variational inference for 2000 iterations, or 5000 for real data, preceded by 250 or 1000 unsupervised-model pretraining iterations.The corresponding pretraining counts are 250 for simulations and 1000 for real data.
  • C.3 Methods details: Training stops early when relative ELBO change falls below 5 × 10^-7.This stopping criterion applies during iFA fitting.
  • C.3 Methods details: Fine-tuning evaluates λ ∈ {0, 0.25, 0.5, 0.75, 1} for simulations and λ ∈ {0, 0.25, 0.5, 0.75, 0.9} for real data.The λ grids differ between simulated and real datasets.
  • C.4 Image representations: Representations use pooled outputs from ViT and DINOv2, and projected image features from CLIP, with each representation having dimension 768.The models are pretrained and obtained through HuggingFace transformers.

D Supplementary figures

The supplementary figures provide a complete view of simulation, pretrained-representation, parameter-recovery, and runtime results. Across scenarios, they visualize covariance transformations, trade-off frontiers, metric sensitivity, iFA consistency, and method runtimes.

  • Pretrained representations: Figure S7 compares CLIP, ViT, and DINOv2 covariance structures under two baselines and the proposed T_ex, T_0, and T_1 transformations.Each panel reports latent–covariate and latent–latent covariances.
  • Simulation scenarios: Figures S8–S11 summarize each scenario through generating covariances, transformed covariances, transformation frontiers, and empirical method comparisons.Scenario 4 omits T_ex because Σ_ZY is not invertible.
  • Metric evaluations: Figures S12–S15 evaluate three proposed metrics and five disentanglement metrics while varying covariate correlations and factor–covariate associations.The proposed metrics are average factor–covariate correlation, factor independence, and average variance explained per factor; disentanglement metrics are D, C, I, SAP, and nSAP.
  • iFA consistency: Figure S16 assesses iFA consistency as observations grow using coefficient recovery, mean squared error, and cosine similarity between recovered and true informed factors.The figure concerns Scenario 1 (PN), with shaded bands showing ±1 s.d. over repetitions.
  • Runtime and methods: Figure S17 compares the runtime of all methods on one simulation setup.The supplementary section also identifies implementation details for comparison methods in Table S5.
Loading 2608.16245v1…