Source-linked AI summary
Projection Pursuit CPCANet for Domain Generalization
Yu-Hsi Chen, Abd-Krim Seghouane
TL;DR
Covariance-based geometric alignment in domain generalization is limited by rank-deficient mini-batch covariance estimation. PP-CPCANet replaces covariance estimation with global Stiefel-manifold projection pursuit and achieves SOTA performance across standard DG benchmarks with stable optimization.
Problem
Domain generalization needs representations that remain effective under unseen distribution shifts, but mini-batch CPCA is limited by rank-deficient covariance estimation and truncated common subspaces.
Method
PP-CPCANet jointly learns a global orthogonal basis and network parameters using Stiefel-manifold optimization with a detached-median L1 projection-pursuit dispersion objective.
Results
PP-CPCANet achieves SOTA performance across multiple standard domain-generalization benchmarks while maintaining stable optimization.
Takeaways & Limitations
Structural analysis supports a single-depth cascade as an effective balance between noise suppression and representation capacity.
Abstract
from arXiv · showhide
Domain Generalization (DG) aims to learn representations robust to distribution shifts. Recent geometric alignment methods, such as CPCANet, extract domain-invariant structures through batch-wise Common Principal Component Analysis (CPCA). However, CPCANet suffers from rank-deficient covariance estimation due to the small-sample-size issue in mini-batch training. To address this limitation, we propose Projection Pursuit CPCANet (PP-CPCANet), a covariance-free framework that learns a global orthogonal basis on the Stiefel manifold and jointly optimizes it with network parameters via the Cayley transform. We further introduce a symmetry-breaking detached-median PP dispersion objective to extract common principal components (CPCs) with dense and robust optimization signals. Experiments on four DG benchmarks show that PP-CPCANet achieves SOTA performance while maintaining stable training.
1. Introduction
Domain generalization seeks representations that remain effective under unseen distribution shifts, but deep CPCA methods face rank-deficient batch covariance estimates. PP-CPCANet addresses this with a covariance-free global orthogonal basis and a robust detached-median L1 projection-pursuit objective.
- Motivation: Domain generalization learns representations from multiple source domains that remain effective on unseen targets despite changing environments and distribution shifts.Deep neural networks often degrade under real-world shifts such as sensor or background variation.
- Method: PP-CPCANet learns a global orthogonal basis through a projection-pursuit dispersion objective on projected features, avoiding covariance estimation.The covariance-free approach retains the ability to discover domain-invariant structures across environments.
- Contributions: PP-CPCANet bypasses batch-wise covariance rank deficiency caused by small sample sizes, preventing representation truncation in deep CPCA models.This directly targets the small-sample-size limitation of mini-batch covariance estimation.
- Contributions: A detached-median L1 projection-pursuit dispersion objective with symmetry-breaking weights enables robust and stable learning.The objective is designed to provide a robust projection-based optimization signal.
2. Related Work
Related work frames domain generalization around learning invariant representations or optimizing robustness to simulated shifts. CPCA-based methods offer geometric alignment through shared bases, but batch-wise estimation is sample-limited, motivating PP-CPCANet’s covariance-free global basis learning and robust dispersion objective.
- Domain Generalization: Domain generalization learns predictors for unseen target domains whose distributions differ from multiple source domains.It uses only source-domain data during training.
- Domain Generalization: Existing DG methods comprise representation learning through invariant features and optimization-based robustness to simulated domain shifts.Representation approaches include adversarial, causal, disentangled, and contrastive methods, while optimization approaches include meta-learning and gradient-based regularization.
- CPCA and CPCANet: CPCA assumes multiple covariance matrices share an orthogonal basis while retaining domain-specific eigenvalues, enabling invariant-structure extraction across domains.CPCANet integrates this principle into deep networks through an unfolded architecture for extracting common principal components.
- CPCA and CPCANet: Batch-wise CPCA remains limited by mini-batch samples, so even dual PCA recovers only a truncated common subspace and may discard geometric information.This limitation motivates bypassing covariance estimation and learning a global orthogonal basis through projection-pursuit optimization.
- Robust Statistical Methods: Standard PCA is outlier-sensitive, motivating robust alternatives based on L1 dispersion or median estimators for domain-shifted observations.Such observations can arise from domain shifts, spurious correlations, or background noise, while directly using median objectives in deep networks is challenging.
3. Projection Pursuit CPCANet
PP-CPCANet avoids mini-batch covariance singularity by jointly learning a global orthogonal basis and network parameters. It combines Cayley-parameterized projections, symmetry-breaking detached-median L1 dispersion, and a progressive cascade with domain-guided modulation in a joint classification objective.
- Global Orthogonal Projection: PP-CPCANet bypasses batch-wise covariance estimation with a globally learnable orthogonal layer jointly optimized with network parameters.The method is designed to address the mini-batch singularity issue inherent to covariance-based optimization.
- Global Orthogonal Projection: The orthogonal basis is parameterized by a skew-symmetric matrix derived from unconstrained parameters and mapped onto the Stiefel manifold through the Cayley transform.A bottleneck first produces a lower-dimensional representation before projection onto the learned basis.
- Robust Projection-Pursuit Objective: A monotonically decreasing weighting scheme breaks rotational symmetry, while detached-median L1 dispersion provides dense and robust optimization signals under domain-shifted outliers.The detached median prevents gradients from propagating through the robust anchor, stabilizing optimization.
- Progressive Cascade and Modulation: The progressive bottleneck cascade narrows latent dimensionality at each depth, using dt+1 = max(dt/2, 16), while domain-guided MLPs predict feature shifts and bounded scaling factors.The cascade follows d1 > d2 > ··· > dT, and modulation is applied before features enter the next depth.
- Training Objective: Training jointly minimizes classification loss and the PP penalty, with the final logits computed from modulated features at the last cascade depth.The PP objective is normalized across cascade depths and source domains, and λppcpca balances the two objectives.
4. Experiments
Experiments evaluate PP-CPCANet on four DG benchmarks under a unified DomainBed protocol and CPCANet optimization settings. PP-CPCANet with VMamba-B achieves the highest average DG accuracy among evaluated models, while incurring lower training overhead than CPCANet; ablations select T = 1 and d1 = 128.
- Datasets: Evaluation covers four DG benchmarks: PACS, VLCS, OfficeHome, and TerraIncognita.PACS, VLCS, and OfficeHome contain 9,991, 10,729, and 15,588 images, respectively, across four domains each.
- Implementation: Training follows DomainBed’s unified protocol and CPCANet’s optimization settings across datasets and backbones.Models use batch size 32 per domain, dropout 0.5, label smoothing 0.1, and learning rates of 5 × 10−5 and 1×10−4 for the backbone and PP-CPCANet modules.
- Main Results: PP-CPCANet with VMamba-B achieves the highest average DG accuracy among all evaluated models.Tables 2 and 3 provide detailed domain-wise results across all benchmarks.
- Main Results: PP-CPCANet consistently incurs lower training overhead than CPCANet in GPU memory and training time.The comparison reports both GPU memory and training time alongside average DG accuracy.
- Hyperparameter Analysis: The best projection dimension is d1 = 128, and the highest accuracy is achieved with the shallow configuration T = 1.Increasing cascade depth to T ≥2 provides no further improvement and degrades performance, so the paper adopts T = 1 and d1 = 128.
5. Conclusion
The paper identifies rank-deficient covariance estimation as a bottleneck for covariance-based geometric alignment and introduces PP-CPCANet to extract domain-invariant common principal components without batch-wise covariance estimation.
- Conclusion: Rank-deficiency bottlenecks covariance-based geometric alignment in deep networks.The limitation is identified in the context of deep-network alignment.
- Conclusion: PP-CPCANet replaces batch-wise covariance estimation with global Stiefel manifold optimization.This covariance-free formulation is designed to address small-sample-size regimes.
- Conclusion: The robust detached-median L1 PP dispersion objective enables domain-invariant CPC extraction with dense, outlier-resistant gradients.The stated optimization properties target small-sample-size settings.