Source-linked AI summary
Efficient Domain Generalization via Common-Specific Low-Rank Decomposition
Vihari Piratla, Praneeth Netrapalli, Sunita Sarawagi
TL;DR
Domain generalization must handle unseen domains without adapting to a fixed target, while models may overfit domain-specific signals. CSD jointly learns common and low-rank domain-specific classifier components, discards the latter at inference, and is reported to generalize better and run considerably faster than existing algorithms. Its analysis establishes identifiability and studies how rank trades off domain-specific noise suppression against generalization.
Problem
Domain generalization seeks zero-shot generalization to multiple unseen domains, but high-capacity networks can overfit domain-specific signals and prior decomposition methods lacked identifiability analysis.
Method
CSD modifies the final softmax layer to learn a common classifier component alongside low-rank domain-specific components, using an orthogonality regularizer and discarding the specific components after training.
Results
CSD generalizes better and is considerably faster than existing algorithms across five speech and image datasets spanning a large range of domains.
Takeaways & Limitations
CSD provides a simple parameter-decomposition approach with identifiable shared parameters and a principled analysis of rank selection for domain generalization.
Takeaways & Limitations
The theoretical analysis assumes the common and domain-specific features are orthogonal, an assumption the paper describes as restrictive.
Abstract
from arXiv · showhide
Domain generalization refers to the task of training a model which generalizes to new domains that are not seen during training. We present CSD (Common Specific Decomposition), for this setting,which jointly learns a common component (which generalizes to new domains) and a domain specific component (which overfits on training domains). The domain specific components are discarded after training and only the common component is retained. The algorithm is extremely simple and involves only modifying the final linear classification layer of any given neural network architecture. We present a principled analysis to understand existing approaches, provide identifiability results of CSD,and study effect of low-rank on domain generalization. We show that CSD either matches or beats state of the art approaches for domain generalization based on domain erasure, domain perturbed data augmentation, and meta-learning. Further diagnostics on rotated MNIST, where domains are interpretable, confirm the hypothesis that CSD successfully disentangles common and domain specific components and hence leads to better domain generalization.
1 Introduction
Domain generalization targets zero-shot prediction on multiple unseen domains, where high-capacity networks can overfit domain-specific signals. CSD addresses this by decomposing classifier parameters and is reported as faster and more accurate than prior approaches.
- 1 Introduction: Domain generalization requires zero-shot generalization from multiple training domains to instances from multiple unseen testing domains.Unlike domain adaptation, it does not adapt to a fixed target domain.
- 1 Introduction: Existing domain-generalization methods learn invariant representations, decompose shared and domain-specific parameters, or use increasingly complex meta-learning procedures.The paper situates CSD among representation-invariance and parameter-decomposition approaches while noting the recent complexity and expense of meta-learning.
- 1 Introduction: CSD decomposes only the final softmax parameters into a common component and a low-rank domain-specific component, with orthogonality regularization.The paper also analyzes the setting theoretically and proves identifiability of the shared parameters.
- 1 Introduction: The paper analytically studies rank as a trade-off between suppressing domain-specific noise and improving domain generalization.This addresses a choice that earlier work treated largely heuristically.
- 1 Introduction: CSD is almost an order of magnitude faster than state-of-the-art meta-learning methods and achieves higher accuracy, especially with many domains.Experiments cover image and speech datasets with 5 to 1000 training domains, alongside rotated-MNIST diagnostics and ablations.
2 Related Work
Domain-generalization research has developed around domain erasure, domain-perturbed augmentation, meta-learning, and parameter decomposition. These approaches respectively target invariant representations, neighboring-domain simulation, cross-domain training objectives, or shared and domain-specific parameters.
- Domain Erasure: Domain-erasure methods reduce divergence between training-domain representations by learning invariant or domain-independent features.Examples use kernels, reconstruction tasks, distribution matching, or projection of superficial features.
- Augmentation: Augmentation methods train classifiers on instances hallucinated from training domains to prepare networks for neighboring domains.Some perturb data along learned directions of domain change, while others apply augmentation to single-domain data.
- Meta-Learning/Meta-Training: Meta-learning methods update parameters using meta-train and meta-test objectives or learn features that remain semantically coherent across those domains.The related-work passage describes this as a recent popular formulation of domain generalization.
- Decomposition: Decomposition methods express network parameters as common plus domain-specific components and retain the common component for inference.Prior work applied this idea to domain adaptation and domain generalization, including low-rank decomposition throughout CNN layers.
3 Our Approach
The approach models domain generalization through common and domain-specific components, then identifies and retains the common classifier for unseen domains. CSD implements this decomposition in the final linear layer with low-rank domain-specific parameters, orthogonality, and a common-component loss.
- Domain generalization assumes common features retain stable label correlations, while domain-specific features vary across domains and generalize less reliably.
- When the common component is not orthogonal to domain-specific components, the domain-generalizing classifier is its projection onto the space orthogonal to Span(Es).Under orthogonality, the common classifier equals ec; otherwise, wc = ec − PEs ec.
- Low rank is treated as a principled design choice because it trades off suppression of domain-specific noise against domain generalization.
- CSD modifies only the final linear layer and adds a common-component prediction loss alongside domain-specific losses.These modifications, together with orthogonality, were reported as instrumental to the method’s performance.
- CSD represents each domain classifier as a common component plus a low-rank domain-specific component, parameterized by Ws and γi.After training, the domain-specific parameters are discarded and only the common classifier is returned.
- In the illustrative linear setting, ERM retained a coefficient of 0.2 on the domain-specific component, whereas CSD reduced it to 0.03 and correctly decomposed the common component.These scaled solutions were obtained with high probability across ten runs.
4 Experiments
CSD is evaluated across image and speech datasets with disjoint training and test domains, varying domain counts and controlled ablations. It generally improves domain-generalization accuracy, does so with modest training overhead, and separates common from domain-specific information.
- Handwritten character datasets: CSD provides significant accuracy gains over ERM, LRD, CG, and MASF across the handwritten-character models and datasets.On LipitK with 76 domains, CSD reaches 87.3 accuracy versus 85.9 for MASF.
- Efficiency: CSD trains 5–10 times faster than MASF, 3–4 times faster than CG, and is only 1.1 times slower than ERM.The comparison indicates limited additional training-time overhead relative to existing methods.
- Speech utterance dataset: CSD outperforms both ERM and CG across speech settings, while standard training can suffice when 1000 training domains cover test-domain variation.The speech comparison excludes MASF because its implementation is available only for image tasks and excludes LRD based on earlier results.
- Rotated MNIST and Fashion-MNIST: On rotated MNIST and Fashion-MNIST, CSD substantially improves out-of-domain accuracy while producing insignificant in-domain improvement, specifically targeting domain generalization.Without batch augmentation, all methods lose 1–4% accuracy, but CSD remains better than MASF.
- How does CSD work?: Figure 1 shows overlapping common-component probability distributions across domains, whereas specialized-component modes are widely separated and reflect rotation degree.The distributions use probabilities assigned to the correct class for training instances, fitted separately by domain.
- Ablation study: Ablations show that the common loss and orthogonality regularizer are both important; combining them significantly outperforms variants lacking either component.Orthogonality addresses bad decomposition, while the common loss supports feature sharing.
- Importance of Low-Rank: Rank k = 1 is best when D ≤100, whereas k = 4 becomes optimal at 200 domains; larger rank can increase estimation noise in the common component.The analysis supports disentangling only domain-specific components observed in training domains.
5 Conclusion
The paper introduces CSD, which decomposes classifier parameters into common and low-rank domain-specific parts, and analyzes its identifiability and rank trade-offs. Across five speech and image datasets, CSD generalizes better and is faster than existing algorithms.
- CSD decomposes classifier parameters into a common component and a low-rank domain-specific component.The method is designed to retain the common part for generalization while separating domain-specific structure.
- The analysis provides identifiability results for CSD and examines how rank trades off domain-specific noise suppression against domain generalization.The paper contrasts this analysis with earlier heuristics-driven treatment of rank.
- CSD generalizes better and is considerably faster than existing algorithms while remaining simple to implement.The paper reports that CSD is almost an order of magnitude faster than state-of-the-art meta-learning methods.
6 Evaluation on PACS dataset
PACS7 evaluates generalization from three image sources to a held-out fourth source, but implementation differences make cross-paper comparisons unreliable. Using the prior implementation and ResNet-18, the paper compares CSD fairly with JiGen and reports similar or slightly better performance.
- PACS7 contains around 10,000 images from seven categories collected from Photo, Art, Cartoon, and Sketch sources.Evaluation trains on three sources and tests on the left-out source, creating a radically different target domain.
- PACS7 comparisons are sensitive to validation splits, augmentation, and optimization settings, rendering results across different implementations unreliable.The passage specifically identifies differences in validation use, data augmentation, learning rate, optimizer, and learning-rate schedule.
- CSD performs almost the same as or slightly better than JiGen in the PACS comparison.The authors avoid comparison with MASF because its reported numbers use an unavailable ResNet-18 implementation and potentially different baselines.
- Table 7 compares JiGen and CSD using ResNet-18, with each column header identifying the target domain.The authors use JiGen’s implementation for a fair comparison.
7 Proof of Theorem 1
Theorem 1’s proof establishes that the construction obtains an optimal low-rank approximation subject to the required shared-direction constraint. It uses an SVD-based decomposition and shows optimality in both Frobenius and operator norms.
- The proof reduces the first two construction steps to finding the best rank-(k + 1) approximation of W whose row space contains 1.This establishes the central approximation property before the final preservation steps.
- Construction preservation: The last two steps preserve the matrix wc1⊤ + WsΓ⊤ while maintaining the required construction properties.This connects the initial approximation argument to the completed matrix produced by the procedure.
- Frobenius-norm optimality: The constrained minimization is rewritten using the centered matrix W − ew1⊤ and its singular value decomposition.The candidate approximation is formed by combining the mean component ew1⊤ with the leading k SVD components.
- Operator-norm optimality: The proof concludes optimality in operator norm for the same constrained approximation problem.The operator-norm argument uses the shared-direction condition and an orthonormal matrix.
- Frobenius-norm optimality: The constructed matrix is shown to be optimal in Frobenius norm among matrices satisfying the conditions in (7).The proof invokes singular values and best rank-i approximations to establish the bound.