PaperlayerView exact revision on arXiv

Source-linked AI summary

The Universal Weight Subspace Hypothesis

Prakhar Kaushik, Shravan Chaudhari, Ankit Vaidya, Rama Chellappa, Alan Yuille

arXiv:2512.05117v2cs.LGcs.AIcs.CV

TL;DR

The paper asks whether networks trained on diverse tasks share low-dimensional structure in their parameters, and analyzes model weights to identify such subspaces. It reports architecture-specific shared subspaces across varied models and tasks, with projected subspaces also supporting strong performance on out-of-distribution tasks.

  • Problem

    The paper examines whether neural networks trained on distinct tasks and data converge to shared low-dimensional parameter subspaces.

    Source: §1 · p. 2, §4 · p. 11

  • Method

    The authors use layer-wise spectral analysis of model weights, retaining leading principal directions to approximate shared subspaces within architectures.

    Source: §3.1 · p. 5, §4 · p. 11

  • Results

    Across tasks, modalities, and training settings, models exhibit architecture-specific shared low-rank subspaces, and projecting onto approximated subspaces supports strong out-of-distribution task performance.

    Source: §4 · pp. 11–12

  • Takeaways & Limitations

    The approximated subspaces support adapting to new tasks without retraining or storing full weights, as well as multi-task learning, scalable fine-tuning, and model merging.

    Source: §4 · p. 12

    The approach relies on pretrained task-specific predictors, which may not be readily available for new tasks, and interpretability of shared directions remains limited.

    Source: §5 · p. 12

Abstract

from arXiv · show

We show that deep neural networks trained across diverse tasks exhibit remarkably similar low-dimensional parametric subspaces. We provide the first large-scale empirical evidence that demonstrates that neural networks systematically converge to shared spectral subspaces regardless of initialization, task, or domain. Through mode-wise spectral analysis of over 1100 models - including 500 Mistral-7B LoRAs, 500 Vision Transformers, and 50 LLaMA-8B models - we identify universal subspaces capturing majority variance in just a few principal directions. By applying spectral decomposition techniques to the weight matrices of various architectures trained on a wide range of tasks and datasets, we identify sparse, joint subspaces that are consistently exploited, within shared architectures across diverse tasks and datasets. Our findings offer new insights into the intrinsic organization of information within deep networks and raise important questions about the possibility of discovering these universal subspaces without the need for extensive data and computational resources. Furthermore, this inherent structure has significant implications for model reusability, multi-task learning, model merging, and the development of training and inference-efficient algorithms, potentially reducing the carbon footprint of large-scale neural models.

1 INTRODUCTION

The introduction proposes that backpropagated networks often learn architecture-specific, layer-wise shared low-rank subspaces across tasks and training conditions. It presents this hypothesis as a parameter-space contribution, distinct from earlier work focused on representations, and outlines evidence, applications, and open questions.

1 INTRODUCTION

  • Networks trained across varied datasets, initializations, hyperparameters, and regularization often learn architecture-specific, layer-wise similar low-rank joint subspaces.The paper names these shared structures Universal Subspaces.Source: §1 · p. 2
  • Earlier work offered indirect or representation-focused universality evidence, whereas this paper claims concrete evidence at the neural-parameter level.The cited examples include recurring circuits and patterns, reusable low-dimensional structures, and recurring eigenvectors in some convolutional layers.Source: §1 · p. 2
  • The analysis spans LoRA adapters, Vision Transformers, and LLaMA models trained on different tasks, datasets, and initializations.The introduction reports approximately 500 Mistral-7B LoRA adapters, 500 Vision Transformers, and 50 LLaMA3-8B models.Source: §1 · p. 2
  • Shared subspaces could support compression, rapid adaptation, and lower computational requirements, while their causes and broader implications remain open.The introduction also raises cross-architecture differences and whether shared biases, capabilities, and failure modes could limit diversity.Source: §1 · pp. 2–3
  • The authors propose learning an approximate shared subspace from available tasks and give conditions for its convergence to the true shared subspace.They also identify reuse of the subspace for adaptation with fewer trainable parameters.Source: §1 · p. 3

2 NOTATIONS, DEFINITIONS AND THEORETICAL ANALYSIS

The theoretical analysis models task predictors in a Hilbert space and studies recovery of their common structure from learned predictors. Its convergence result separates error from having finitely many tasks from per-task estimation error.

2 NOTATIONS, DEFINITIONS AND THEORETICAL ANALYSIS

  • The theoretical model uses Hilbert-space predictors, while the experiments analyze practical large-scale transformers and LoRA variants.The paper motivates this modeling choice as standard for studying generalization and inductive bias in modern deep architectures.Source: §2 · p. 3
  • The analysis asks when empirical operators built from learned task predictors concentrate around the population second-moment operator and their top-k subspace converges to its true counterpart.The two error sources are finite-task variability and within-task estimation noise.Source: §2 · p. 3
  • Tasks are modeled as draws with ground-truth and learned predictors in a separable Hilbert space, and the population operator’s top-k eigenspace defines the shared subspace.The analysis uses a second-moment operator rather than centered covariance, so its top eigenspace may include the mean predictor direction.Source: §2 · pp. 3–4
  • The assumptions require bounded ground-truth predictors with finite second moment and bounded effective rank, plus accurate per-task predictor estimates.The estimation condition is expressed through sample size and hypothesis-space complexity; the text gives strongly convex regularized ERM in an RKHS as an example.Source: §2 · p. 4
  • With probability at least 1 − δ, the empirical operator and learned top-k subspace converge to their population counterparts under the stated assumptions.The rates reflect averaging across tasks and per-task estimation errors; a larger eigengap makes subspace recovery more stable.Source: §2 · p. 4

3 ANALYSIS

Across diverse training tasks and configurations, the analysis finds architecture-specific, layer-wise low-rank subspaces shared by independently trained models. It uses spectral decomposition to examine these subspaces and tests their representation, merging, and reuse across CNNs, adapters, and pretrained weights.

3 ANALYSIS

  • Architecture-specific, layer-wise universal subspaces appear robust across training from scratch, full finetuning, and low-rank adaptation, despite differences in initialization, modality, and data.The proposed ideal universal subspace is not formally verified, and subspace fidelity correlates with the quantity and quality of available models.Source: §3 · p. 5

3.1 ANALYSIS METHODOLOGY

  • Order 1–2 HOSVD retains, for each mode, the smallest number of left singular vectors whose cumulative explained variance is at least τ, yielding a truncated core from the zero-centered tensor.Reconstruction adds the elementwise mean to the truncated core multiplied mode-wise by the factor matrices.Source: §3.1 · p. 5

3.2 RESULTS FROM JOINT SUBSPACES’ ANALYSIS

  • The analysis includes eight applications across image classification, language understanding, text-to-image generation, and model merging, using public models without training costs for subspace extraction.Spectral decomposition can run on CPUs; the reported analyses and experiments used one Nvidia A5000 GPU.Source: §3.2 · pp. 5–6

3.2.1 LOWER-RANK JOINT SUBSPACES IN CNNS, LORA AND FINETUNED MODELS

  • Across five ResNet-50 models trained on disjoint datasets, a shared low-rank structure spans tasks, and projection preserves competitive performance relative to full fine-tuning.The study notes that the small number of models may yield an under-approximation; Figure 2 reports majority information in 16 or fewer directions across layers.Source: §3.2.1 · p. 6, Figure 2 · p. 6
  • Across Mistral LoRAs and SDXL adapters, universal projections preserve seen- and unseen-task performance and generation style, with 19× lower LoRA memory use and occasional CLIP-score gains.For SDXL, the CLIP-score improvement over individual LoRAs is reported in some cases and is possibly due to denoising effects.Source: Figure 3 · p. 7, §3.2.1 · p. 8, Figure 5 · p. 8
  • The universal-subspace merging approach achieves higher average accuracy than six gradient-free baselines while reducing parameter count, without iterative tuning or validation data.Optional finetuning is possible when data are available; the authors note they did not optimize the merging process.Source: §3.2.1 · p. 9, Table 2 · p. 9

3.3 LOW RANK SHARED UNIVERSAL SUBSPACES IN CLASSICAL WEIGHTS

  • Across diverse pretrained weights, a few spectral directions capture dominant structure; projecting five unseen ViTs into a 16-dimensional subspace causes no significant accuracy drop.The 500-model ViT representation excludes task-variable first and last layers and requires 100× less memory; analyses also include LLaMA-8B, GPT-2, and Flan-T5 models.Source: §3.3 · p. 10, Figure 6 · p. 10

3.3.1 FINDING UNIVERSAL SUBSPACES AND APPLYING THEM TO FUTURE TASKS

  • For new tasks, freezing shared directions and learning task-specific coefficients reduces trainable parameters; the reported universal subspace reduces ViT and LLaMA model storage by more than 100×.The experiments cover ViT image classification and six GLUE tasks, omitting MNLI and QQP.Source: §3.3.1 · p. 11

4 DISCUSSION

Across architectures and settings, the paper reports shared, architecture-specific low-rank subspaces in trained networks. It describes these findings as a basis for reusing principal directions, while noting that the mechanisms behind their emergence remain unresolved.

4 DISCUSSION

  • The analysis reports layer-level shared low-rank subspaces across diverse tasks, modalities, initializations, and training settings.The paper says this pattern occurs in fully finetuned and LoRA-based models, among other configurations.Source: §4 · pp. 11–12
  • Reusing layer-wise principal directions and learning lightweight task coefficients could reduce memory, compute, and engineering overhead for adaptation and deployment.The paper also identifies possible financial and environmental benefits, while presenting these as implications of the findings.Source: §4 · p. 12
  • The precise mechanisms behind shared subspace emergence remain open; proposed contributors include spectral bias, architectural inductive biases, and gradient-based optimization.The paper presents these factors as hypotheses, not established explanations.Source: §4 · p. 12

5 LIMITATIONS AND FUTURE WORK

The paper identifies limited interpretability and dependence on pretrained task-specific predictors as boundaries of its current analysis. It also describes task arithmetic within the shared-subspace framework as non-trivial and requiring further investigation.

5 LIMITATIONS AND FUTURE WORK

  • Interpretability of the shared subspace and its principal directions remains limited, in part because examining directions layer by layer is cumbersome.The paper identifies this as an area for future research.Source: §5 · p. 12
  • The current approximation approach relies on pretrained task-specific predictors, which may not be readily available for new tasks.The paper proposes exploring model-independent methods, potentially derived directly from data.Source: §5 · p. 12
  • Task arithmetic within the shared-subspace framework is non-trivial because the cited conditions rely on localized eigenfunctions.The paper says this warrants further investigation.Source: §5 · p. 12

A.1 RELATED WORK

Prior work offers related indications of shared structure, but the paper distinguishes its contribution as a scalable empirical analysis of shared parametric subspaces. It connects that analysis to a framework for reusing subspaces in adaptation and model merging.

A.1 RELATED WORK

  • Earlier NTK and mechanistic-interpretability work supports intuitions about shared structure but does not provide a unified, scalable framework across architectures, tasks, and modalities.The paper contrasts its parameter-level evidence with prior work focused on functional spaces or recurring representations.Source: §A.1 · p. 17
  • The figure reports rapid spectral decay across Mistral-7B LoRAs, Vision Transformers, and LLaMA-8B models, which the paper treats as evidence for joint low-dimensional subspaces.Its accompanying discussion proposes compression and coefficient-based tuning as implications, while raising questions about recovering the subspace without extensive data.Source: Figure 7 · p. 18
  • Lottery-ticket, mode-connectivity, and convolutional-filter studies are presented as evidence of sparse, connected, or recurring structures in neural networks.The passage also cites recurring eigenvectors in convolutional layers across networks trained on different datasets.Source: §A.1 · p. 18
  • The paper positions its contribution as a principled, empirically validated method for finding and using parametric subspaces across architectures, tasks, and modalities.It describes prior studies as suggestive but limited in focus, application, and analysis.Source: §A.1 · pp. 18–19

A.2 THEORETICAL ANALYSIS

The theoretical analysis applies generalization and concentration bounds to projection onto a shared subspace. It assumes bounded task predictors and projections in an RKHS, and states a high-probability theorem under specified assumptions.

A.2 THEORETICAL ANALYSIS

  • The analysis bounds squared error between each task function and its projection onto the shared subspace, assuming both lie in an RKHS with norm at most B.The text uses a kernel bound to bound pointwise squared loss and establish conditions for PAC-style generalization bounds.Source: §A.2 · p. 19
  • The proof invokes matrix Bernstein bounds for independent, mean-zero, bounded self-adjoint operators as part of its concentration argument.The supplied proof passages also refer to averaging, operator norms, and a union bound with Davis–Kahan.Source: §A.2 · pp. 19–21
  • Theorem A.3 states a probability guarantee of at least 1 − δ over tasks and per-task samples under Assumptions 2.3–2.4.The supplied theorem passage does not include the bound's conclusion.Source: §A.2 · p. 20

B UNIVERSAL SUBSPACE ANALYSIS

The analysis constructs shared low-rank subspaces from model weights and defines a residual secondary subspace for comparison. It also specifies ways to select the principal-component rank and limits which layers and parameters are analyzed.

B UNIVERSAL SUBSPACE ANALYSIS

  • The analysis covers matrix weights in most layers, excluding initial and final layers that may vary with input shape, task, and loss.It focuses on linear or fully connected matrix weights; the authors state that results extend to other neural parameters.Source: B UNIVERSAL SUBSPACE ANALYSIS · p. 21
  • The secondary subspace is the residual after removing the top k directions, is orthogonal to the universal subspace, and serves as a control.For tractability, the analysis uses top components beyond rank k rather than a full SVD; lower components are treated as typically noise-capturing.Source: B UNIVERSAL SUBSPACE ANALYSIS · p. 21
  • The rank k can be chosen by scree-plot inspection, an explained-variance threshold below 0.01, or an optimal HOSVD singular-value threshold.The passage describes components with explained variance close to zero (<.01) as secondary and discardable.Source: B UNIVERSAL SUBSPACE ANALYSIS · p. 21

B.1 LOWER RANK SHARED UNIVERSAL SUBSPACES WITHIN LOW RANK ADAPTATION (LORA) MODELS

The LoRA analysis extracts layer-specific shared subspaces by combining principal directions from adapters, then tests their spectral concentration and reconstruction performance. It reports compact shared structure in Mistral and SDXL experiments, including performance retention and memory savings.

B.1 LOWER RANK SHARED UNIVERSAL SUBSPACES WITHIN LOW RANK ADAPTATION (LORA) MODELS

  • Layer-specific LoRA subspaces are extracted by concatenating rank vectors from adapters, centering their features, and computing principal directions for matrices B and A.The mode-1 variant is mathematically equivalent to PCA; the same calculation can use the BA matrix without changing the outcome, but at greater computational cost.Source: §B.1 · p. 22
  • Across 500 Mistral LoRAs, the top 16 principal directions consistently capture most variance, supporting a shared low-dimensional subspace.The aggregate plot spans all layers; the layerwise plots show key, query, and value matrices, with only the top 128 directions visualized.Source: Figure 8 · p. 22, §B.1 · p. 25, Figure 9 · p. 27
  • The Mistral experiment analyzes 500 LoRAs trained on distinct Natural Instructions tasks using Mistral-7B-Instruct-v0.2 as their base model.The evaluation compares reconstructed weights on seen and unseen tasks.Source: §B.1 · pp. 22, 26
  • Projecting LoRA weights into the universal subspace retains high performance on 5 seen and unseen tasks, while the secondary subspace causes a sharp drop; storage is 19× more memory efficient.The universal representation eliminates the need to store all 500 LoRAs.Source: §B.1 · p. 26
  • For SDXL, projecting individual LoRAs into one universal subspace matches their visual quality and style nuances, and the reported CLIP scores exceed those of individual LoRAs.The subspace uses 16 components; each evaluated adapter is reconstructed from a subspace extracted using the other available adapters.Source: §B.1 · p. 28

B.2 LOW RANK SHARED UNIVERSAL SUBSPACES IN CLASSICAL WEIGHTS

The classical-weight analysis finds low-dimensional shared structure in ViT and LLaMA models, extending the evidence beyond adapters. The analysis is more constrained and noisier than the LoRA setting, but leading components still show consistent structure.

B.2 LOW RANK SHARED UNIVERSAL SUBSPACES IN CLASSICAL WEIGHTS

  • Across approximately 500 ViT models, the top 16 directions capture most variance, and 16-component reconstructions perform competitively on selected IID and OOD models.The evaluated OOD models were excluded from the subset used to construct the universal subspace; models were used as-is without curation or training-data access.Source: Figure 10 · p. 28, §B.2 · p. 29
  • Convolutional weights are excluded from explicit analysis, which the authors say generalizes naturally to them through their stated relation to fully connected layers.The passage also notes prior limited-scope work on Gabor-like filters.Source: §B.2 · p. 29
  • Classical-weight analysis costs more computation, has fewer well-trained models and greater model-quality variation, and yields noisier subspaces despite consistent leading structure.The authors attribute the greater quality variance to training from scratch without the strong pretrained initialization used by LoRA.Source: §B.2 · p. 29
  • Across 50 diverse LLaMA-3-8B models, variance concentrates in a few leading components, while most layerwise eigenvalues fall below 0.001.The models span medical, multilingual-dialogue, and general-purpose domains; the plots show only leading components.Source: §B.2 · pp. 33–34, Figure 12 · p. 34
  • Flan-T5 models fine-tuned on individual GLUE datasets show trends similar to the other analyses, with low-rank subspaces emerging across tasks and datasets.Source: §B.2 · p. 34

B.3 ABLATING NUMBER OF MODELS AND SUBSPACE EFFECTIVENESS

The ablation examines how the number of models affects universal-subspace estimation and reports diminishing returns as more models are added. The authors relate saturation to task count while noting that comprehensive ablations are constrained by data and compute requirements.

B.3 ABLATING NUMBER OF MODELS AND SUBSPACE EFFECTIVENESS

  • As task count T increases, Theorem 2.5 gives convergence of the shared subspace to the true subspace at O(1/T).The passage describes this rate as indicating increasingly effective coverage as T increases.Source: §B.3 · p. 34
  • In Mistral OOD evaluation, adequate principal components emerge quickly as more models contribute to the universal subspace, with diminishing returns.The ablation uses nine randomly chosen OOD tasks and reconstructs their coefficients analytically; Table 12 reports the evaluation.Source: Table 12 · p. 35, §B.3 · p. 35
  • The model count needed for saturation depends on model quality, training-data diversity, and architecture, while exhaustive ablations exceed practical time, data, and compute constraints.Source: §B.3 · p. 35

C FINDING UNIVERSAL SUBSPACES AND APPLYING THEM TO FUTURE TASKS

Layerwise scree plots show that 50 LLaMA-3-8B models concentrate explained variance in a few leading directions. The section also describes coefficient-only adaptation and the broader efficiency implications of shared subspaces.

C FINDING UNIVERSAL SUBSPACES AND APPLYING THEM TO FUTURE TASKS

  • For image classification and GLUE, the shared-subspace experiments train only coefficients, using subspaces extracted from other models in each subset.Image classification uses k = 4 and 8 epochs; GLUE uses 16–32 components and 30–80 epochs.Source: C FINDING UNIVERSAL SUBSPACES AND… · p. 35
  • Universal-subspace extraction can run on CPU, while GPU parallelization of layerwise extraction would increase calculation speed.The experiments used a single A5000 GPU and a CPU with 8 workers.Source: C FINDING UNIVERSAL SUBSPACES AND… · p. 35
  • Across 50 LLaMA-3-8B models, explained variance concentrates in a few leading principal directions, while later eigenvalues approach zero.The plots present a truncated view of the possible principal directions.Source: Figure 13 · p. 36

D DISCUSSION AND BROADER IMPACT

  • The authors say shared low-dimensional subspaces permit adapting models by learning a small number of coefficients rather than retraining or storing full weights.They connect this approach to multi-task learning, model merging, and scalable fine-tuning.Source: D DISCUSSION AND BROADER IMPACT · p. 37
  • The authors present reduced computational overhead for reuse and extension as a way to address financial and environmental costs and lower hardware and energy requirements.They connect these reductions to more accessible AI for communities with limited compute infrastructure.Source: D DISCUSSION AND BROADER IMPACT · p. 37

Related papers

Loading 2512.05117v2…