PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Weight Space Learning: Understanding, Representation, and Generation

Xiaolong Han, Zehong Wang, Bo Zhao, Binchi Zhang, Jundong Li, Damian Borth, Rose Yu, Haggai Maron, Yanfang Ye, Lu Yin, Ferrante Neri

arXiv:2603.10090v1cs.LG

TL;DR

Neural network weights are an increasingly rich but insufficiently organized domain for analyzing and comparing pretrained models. This survey unifies Weight Space Learning into understanding, representation, and generation, showing how these approaches expose model-space structure and support model-level analysis and synthesis.

  • Problem

    Pretrained weights form a growing source of information, but existing weight-space developments lack a cohesive structure for studying them systematically.

    Source: §1 · pp. 4–5

  • Method

    The survey organizes Weight Space Learning into weight-space understanding, representation, and generation as complementary dimensions.

    Source: §1 · p. 5, §8 · p. 38

  • Results

    Weight-space methods expose global model-space structure, enabling semantic model synthesis, interpolation, recombination, and predictive embeddings for automated model analysis.

    Source: §3.2.1 · p. 18, §4.3 · p. 28

  • Takeaways & Limitations

    Weight Space Learning treats models as structured objects that can support model-level reasoning, comparison, transfer, and direct weight generation.

    Source: §4.3 · p. 29, §8 · p. 38

    Most WSL methods target small or medium-sized networks, while scaling to large models is computationally prohibitive because weight spaces grow and contain complex dependencies.

    Source: §7.2 · p. 37

Abstract

from arXiv · show

Neural network weights are typically viewed as the end product of training, while most deep learning research focuses on data, features, and architectures. However, recent advances show that the set of all possible weight values (weight space) itself contains rich structure: pretrained models form organized distributions, exhibit symmetries, and can be embedded, compared, or even generated. Understanding such structures has tremendous impact on how neural networks are analyzed and compared, and on how knowledge is transferred across models, beyond individual training instances. This emerging research direction, which we refer to as Weight Space Learning (WSL), treats neural weights as a meaningful domain for analysis and modeling. This survey provides the first unified taxonomy of WSL. We categorize existing methods into three core dimensions: Weight Space Understanding (WSU), which studies the geometry and symmetries of weights; Weight Space Representation (WSR), which learns embeddings over model weights; and Weight Space Generation (WSG), which synthesizes new weights through hypernetworks or generative models. We further show how these developments enable practical applications, including model retrieval, continual and federated learning, neural architecture search, and data-free reconstruction. By consolidating fragmented progress under a coherent framework, this survey highlights weight space as a learnable, structured domain with growing impact across model analysis, transferring, and weight generation. We release an accompanying resource at https://github.com/Zehong-Wang/Awesome-Weight-Space-Learning.

1. Introduction

Weight Space Learning treats pretrained neural-network weights as a structured, learnable data modality and organizes prior work into understanding, representation, and generation. This survey provides a unified taxonomy and roadmap for studying these dimensions and their applications.

Motivation

  • Pretrained weights encode task knowledge, architectural biases, optimization dynamics, and learning regularities, motivating weight space as a new learnable data modality.Unlike input features, weights describe the learners of data.Source: §1 · p. 4

Weight Space Learning

  • Weight Space Learning treats model parameters as structured objects that can be analyzed, represented, and generated rather than merely optimized.Recent studies explore all three directions [Erkoç et al., 2023, Navon et al., 2023, Schürholt et al., 2022a].Source: §1 · p. 4

Research Gap

  • The survey addresses conceptual fragmentation by unifying diverse terminology and prior work under a coherent research paradigm.Existing studies span symmetry analysis, model compression, retrieval, and generative modeling, often using inconsistent names and frameworks [Zhou et al., 2023a, Ashkenazi et al., 2023].Source: §1 · pp. 4–5

Taxonomy

  • The taxonomy defines Weight Space Understanding, Weight Space Representation, and Weight Space Generation as complementary dimensions of WSL.WSU studies intrinsic structures and principles; WSR learns weight embeddings or descriptors; WSG synthesizes weights through hypernetworks or generative models.Source: §1 · p. 5, Figure 2 · p. 5

Survey Scope

  • The survey further covers applications, evaluation resources, and open directions, including model retrieval, function prediction, model editing, weight generation, benchmarks, and datasets.Its later sections address practical application scenarios, benchmarks, datasets, emerging trends, challenges, and future research directions.Source: §1 · p. 5

2. Weight Space Understanding

Weight Space Understanding treats neural parameters as a structured domain shaped by symmetries, redundancies, geometry, and topology rather than as unstructured vectors. It connects these structures to functional equivalence, optimization, compression, augmentation, and open challenges in making weight-space geometry predictive.

2.1. Structural Foundations

  • Symmetry organizes weight space into equivalence classes and structured transformations, explaining functional redundancy, optimization degeneracy, and generalization while enabling model-level learning.WSU studies parameter symmetries independently of datasets or objectives, including invariance and equivariance as induced functional relations.Source: §2 · p. 7, §2.1 · p. 8

2.1.1. Functional Invariance

  • Invariance maps distinct parameterizations to the same function, motivating symmetry-invariant representations, compression, mode connectivity, model merging, and weight-space interpolation.Neuron permutations, positive scaling, and logit translations are examples; these symmetries create flat valleys and functional equivalence classes.Source: §2.1.1 · pp. 8–9

2.1.2. Functional Equivariance

  • Equivariance makes parameter transformations induce predictable output transformations, supporting symmetry-aware meta-models, representations, model editing, and generative synthesis across architectures.Examples include rotations, sign flips, and attention-head transformations; DWSNets, NFNs, UNFs, and EQUIGEN operationalize these principles.Source: §2.1.2 · p. 10

2.2. Technique-Level Use Cases

  • WSU translates symmetry orbits and equivalence classes into practical strategies for compression, optimization, and data augmentation that improve efficiency, robustness, or generalization.These applications bridge theoretical weight-space structure with learning methods that operate on functionally meaningful configurations.Source: §2.2 · p. 11

2.2.1. Model Compression

  • Symmetry-guided compression removes functionally redundant parameters while preserving performance, using graph merging, factorized representations, pruning, and low-rank decomposition.These methods also expose how architectural design creates redundancy in overparameterized networks.Source: §2.2.1 · p. 11

2.2.2. Model Optimization

  • Symmetry reshapes optimization as navigation over connected manifolds of equivalent minima, motivating orbit-based, invariant, rebalancing, and quotient-space methods.Examples include Symmetry Teleportation, Path-SGD, G-SGD, projection-based updates, and explanations of low-loss mode connectivity.Source: §2.2.2 · pp. 11–12

2.2.3. Weight Space Augmentation

  • Weight-space augmentation uses symmetry-preserving transformations and aligned interpolation to create diverse, semantically consistent models, especially for data-scarce implicit neural representations.Weight Space MixUp, permutation augmentation, equivariant mappings, and EQUIGEN support smoother function families and few-shot generation.Source: §2.2.3 · p. 12

2.3. Discussion and Perspective

  • WSU provides a coherent foundation for interpreting redundancy, optimization, and generalization, but must address incompletely characterized symmetries, scalability, heterogeneous architectures, and predictive geometric principles.Large practical networks introduce complex parameter couplings and data-dependent manifolds; quotient-space computation can involve intensive graph matching, while measurable links to generalization, robustness, and adaptability remain open.Source: §2.3 · pp. 12–13

3. Weight Space Representation

Weight Space Representation (WSR) learns compact, transferable embeddings of neural network parameters to capture model semantics, behavior, or function. These representations support model-level inference and practical tasks while connecting weight-space structure with data-independent model analysis.

Representation formulation

  • WSR maps neural network weights θ∈Θ to low-dimensional embeddings z that capture structural or functional model properties.The representation function ϕ: Θ → ℝ^d supports reasoning directly in model space.Source: §3 · p. 13

Downstream objectives

  • The learned embeddings enable performance prediction, similarity retrieval, model editing, and other downstream model-level objectives.A meta-model formed by ϕ and g learns from networks rather than data, bridging model understanding and functional inference.Source: §3 · p. 13

WSR paradigms

Challenges and scope

  • WSR must address architectural heterogeneity and extreme dimensionality through efficient structure-aware architectures and regularization that preserve weight-space structure while scaling [Kofinas et al., 2024a, Lim et al., 2024, Zhou et al., 2024] [Zhao et al., 2025b] [Navon et al., 2023, Zhou et al., 2023a] [Andreis et al., 2024, Ashkenazi et al., 2023].Different architectures organize and scale parameters differently, while modern networks may contain millions of weights, complicating compact, semantically consistent representations.Source: §3 · p. 14

Methodological scope

  • WSR methodologies span explicit statistical features, learning-based formulations, and probing-driven approaches that support model retrieval, property prediction, and implicit neural representation.This positions WSR as a link between theoretical weight-space structure and data-independent model analysis.Source: §3 · p. 14

3.1. Representation Approaches

Weight Space Representation learns low-dimensional embeddings of neural networks through model-based encoders over parameters or model-free encoders over functional behavior. Model-based methods progress from direct parameter modeling to handcrafted symmetry-aware and graph-based architectures, while model-free probing provides architecture-agnostic, symmetry-invariant behavioral representations.

3.1. Representation Approaches

  • WSR defines an embedding 𝜙: Θ → ℝ^d and divides methods into model-based approaches over weights and model-free approaches over predictive behavior.The taxonomy is illustrated in Figure 5, with model-based methods encoding weight values and connectivity and model-free methods learning from responses to reference probes.Source: §3.1 · p. 14, Figure 5 · p. 15

3.1.1. Model-based Representation

  • Model-based representation evolves from symmetry-agnostic encoders to handcrafted symmetry-aware architectures and graph-based frameworks that automatically capture structural symmetries.This progression shifts from direct parameter regression toward automatic encoding of architectural invariances.Source: §3.1.1 · pp. 14, 16
  • Symmetry-agnostic encoders directly map weights to embeddings but often fail across architectures because they ignore permutation and scaling invariances.Examples include statistical predictors, CNNs, MLPs, boosted trees, and transformer-based hyper-representations trained on model weights.Source: §3.1.1 · pp. 14–15
  • Symmetry-aware methods impose equivariance or approximate invariance through shared weights, group-orbit constructions, attention, regularization, and expanded symmetry groups, but handcrafted designs limit scalability across heterogeneous architectures.DWSNets and NFN address permutation symmetry, later work incorporates scaling, sign-flipping, rotational, and broader architectural symmetries.Source: §3.1.1 · pp. 15–16
  • Graph-based methods model networks as directed graphs with biases as node features and weights as edge features, using GNN message passing to learn architecture-agnostic, symmetry-aware embeddings without manual specification.Neural Graph, Graph Metanetworks, Scale-GMN, and related work propagate information over computational dependencies and extend the approach across parameterizations and architectures.Source: §3.1.1 · p. 16

3.1.2. Model-free Representation

  • Model-free methods infer latent descriptors from finite probe responses, making representations architecture-agnostic, naturally invariant to parameter symmetries, and usable when weights are inaccessible.Probe-based methods range from static logit probing to learned probes, hidden-layer mixtures of experts, GNN-integrated activations, and adaptive interactive probes for RNNs.Source: §3.1.2 · pp. 16–17

3.2. Technique-Level Use Cases

Weight Space Representation embeds neural networks into compact latent vectors, enabling model-level reasoning without original training data. These embeddings support behavior prediction, model retrieval, and efficient functional editing.

3.2. Technique-Level Use Cases

  • WSR maps network parameters to compact latent representations that capture functional and structural information, enabling prediction, retrieval, and editing without training datasets.This reframes weight space as a functional manifold where model structure and behavior align, supporting scalable evaluation and reuse.Source: §3.2 · p. 17

3.2.1. Behavior Prediction

  • Weight embeddings encode predictive signals about capacity, optimization history, and task fit, enabling forecasts of accuracy, hyperparameters, generalization gaps, and learning dynamics.Behavior prediction formulates model assessment as regression: ŷ=ψ(ϕ(θ)). Early works include DCM [Eilertsen et al., 2020] and Unterthiner et al. [2020], with later studies scaling to model zoos and undocumented Hugging Face models.Source: §3.2.1 · pp. 17–18

3.2.2. Model Retrieval

  • WSR enables data-independent model matching by retrieving networks with functionally aligned embeddings, including large-scale search across thousands of Hugging Face models.ProbeX [Horwitz et al., 2025a], ProbeLog [Kahana et al., 2025b], and Horwitz et al. [2025b] operationalize embedding-level model discovery.Source: §3.2.2 · p. 18

3.2.3. Model Editing

  • WSR supports targeted behavioral modification through small, structured latent perturbations that avoid full fine-tuning while preserving overall model integrity.Dravid et al. [2024] and Schürholt et al. [2024] demonstrate bias removal and capability enhancement; permutation-equivariant weight transformations also edit INRs and generate data without full retraining [Kofinas et al., 2024a, Lim et al., 2024, Zhou et al., 2023a,b].Source: §3.2.3 · p. 18

3.3. Discussion and Perspective

Weight Space Representation (WSR) embeds neural networks as geometric data points in structured latent spaces, enabling model-level reasoning, comparison, manipulation, and data-free generalization. Its future depends on overcoming cross-architecture, data, and evaluation barriers while integrating predictive, self-supervised, and generative representations.

Strengths and Contributions

Limitations and Open Challenges

  • High dimensionality, architectural heterogeneity, uneven pretrained-model repositories, and missing standardized benchmarks constrain WSR’s transferability and evaluation.Function prediction accuracy alone does not measure symmetry preservation or transferability, motivating theoretical formalization and empirical grounding [Falk et al., 2025a, Schürholt et al., 2022b].Source: §3.3 · p. 19

Outlook

  • The next frontier is predictive and generative WSR combining cross-architecture embeddings, self-supervision over implicit symmetries, and weight synthesis.These directions aim to connect weight space understanding and generation, enabling networks to be compared, composed, and created within one representational space.Source: §3.3 · p. 19

4. Weight Space Generation

Weight Space Generation (WSG) treats neural-network weights as generative objects, learning mappings from latent variables to coherent, diverse, and task-aligned model parameters. It enables efficient synthesis and adaptive generation across applications while facing challenges from weight-space dimensionality, data dependence, and cross-architecture stability.

Scope

  • WSG extends weight-space learning from understanding and representation to creation, positioning neural networks as generated objects rather than only optimized or represented parameters.Its broader scope is to learn generative processes over parameter distributions.Source: §4 · pp. 19–20

Representative paradigms

  • Hypernetworks generate target-model parameters from conditioning prompts using end-to-end downstream-task training, while generative models iteratively denoise weight-space noise for conditional synthesis.These paradigms support fast weight generation and diverse, task-aligned configurations without explicit gradient-based optimization.Source: Figure 6 · p. 20

Core formulation

  • WSG learns mappings from latent codes to model weights, capturing architectural symmetries, scaling relations, and functional similarity to generate coherent, diverse, task-aligned networks.A latent code z ∈ Z is transformed into parameters θ = G_ϕ(z) for a target model.Source: §4 · p. 20

Motivation and applications

  • WSG enables efficient model synthesis without full retraining and adaptive generation conditioned on tasks, data domains, or performance objectives.Applications include meta-learning [Nava et al., 2023b, Zhao et al., 2020], continual learning [Li et al., 2025, von Oswald et al., 2020], federated learning [Lai et al., 2025, Li et al., 2024a, Shamsian et al., 2021], and model fusion [Tang et al., 2025, Tian et al., 2025].Source: §4 · p. 20

Limitations

  • WSG remains challenged by extreme weight-space dimensionality, strong dependence of learned priors on data distributions, and unresolved stability and generalization across architectures.These limitations hinder efficient sampling and reliable generation across changing settings.Source: §4 · p. 20

4.1. Generation Approaches

Weight-space generation comprises two paradigms: hypernetworks map conditioning signals to target weights, while generative models learn distributions over trained weights for diverse sampling. The section contrasts their mechanisms and highlights unifying controllability with diversity as a key frontier.

4.1. Generation Approaches

  • The key frontier is to combine hypernetworks’ controllable adaptation with generative models’ diversity and scalability for precise adaptation and open-ended model synthesis.Diffusion-based methods exemplify distribution-level generation by denoising from noise, while conditioning can reintroduce task-specific control and support larger models and adaptation modules.Source: §4.1 · p. 21, §4.1.2 · pp. 25–26
  • Weight-space generation splits into hypernetworks that produce target weights and generative models that sample from learned distributions over trained weights.Hypernetworks use auxiliary networks conditioned on signals such as tasks or architectures, whereas generative models learn the distribution of model weights themselves.Source: §4.1 · p. 21

4.1.1. Hypernetworks

  • Hypernetworks map task, architecture, class, or noise conditions to target parameters, enabling model instantiation or adaptation through a single forward pass.Task loss backpropagates through the generated target model into the hypernetwork, forming a bi-level optimization scheme; shared generation can reduce optimization costs.Source: §4.1.1 · pp. 21–22
  • Hypernetwork conditioning has progressed from stochastic or handcrafted prompts to structured graph, task, semantic, image, and feature inputs, while outputs have expanded from full tensors to chunks, adapters, LoRA modules, and residual updates.These designs broaden hypernetworks from direct parameter emitters into scalable controllers for adaptation, specialization, personalization, and targeted editing [Krueger et al., 2017, Pawlowski et al., 2017, Yin et al., 2022, Mahabadi et al., 2021, Ruiz et al., 2024].Source: §4.1.1 · pp. 22–23

4.1.2. Generative Models

  • Generative model approaches learn weight-space distributions independently of explicit prompts, supporting diverse exploration and four families: autoencoder, adversarial, autoregressive, and iterative-denoising methods.These families respectively use latent reconstruction, minimax synthesis, sequential token generation, or progressive denoising to create new weights.Source: §4.1.2 · pp. 23–25

4.2. Technique-Level Use Cases

Weight Space Generation (WSG) supports adaptive specialization, efficient optimization, model merging, initialization, training acceleration, and data synthesis by generating structured weights on demand. These use cases shift adaptation and generation from iterative parameter updates or raw arithmetic toward conditioned, distribution-aware weight synthesis.

4.2. Technique-Level Use Cases

  • Overall, WSG provides a unified mechanism to synthesize, compose, initialize, and generate models or data on demand for changing tasks, domains, and conditions.Its applications span conditional and real-time generation, model merging, initialization, training acceleration, and data synthesis.Source: §4.2 · p. 26

4.2.1. Conditional Weight Generation

  • Conditional WSG synthesizes task-, domain-, or semantic-specific weights, enabling flexible adaptation and novel-category expansion without full retraining [Yin et al., 2022; Nava et al., 2023a; Tian et al., 2025; Li et al., 2025].The shared mechanism is a conditioned mapping from semantic or domain signals to weights, supporting data-efficient control of model behavior.Source: §4.2.1 · p. 26

4.2.2. Real-Time Weight Optimization

  • WSG bypasses per-instance gradient optimization by synthesizing weights in a single forward pass, reducing latency for streaming, interactive, and real-time adaptation [Nirkin et al., 2021; Alaluf et al., 2022; Zhang et al., 2025d].These methods generate decoder, generator-modulation, or classifier-head weights for new tasks or environments without full retraining.Source: §4.2.2 · pp. 26–27

4.2.3. Model Merging

  • WSG enables model merging in latent or hyper-representation space, preserving structural relationships and avoiding destructive interference from naive parameter averaging [Tang et al., 2024, Wortsman et al., 2022].Shared representations, generative recombination, and equivariant alignment address permutation symmetry and support merged models that generalize beyond individual source networks [Schürholt et al., 2022a; Zeng et al., 2025; Navon et al., 2024; Navon et al., 2023].Source: §4.2.3 · p. 27

4.2.4. Weight Initialization

  • WSG learns task-aware initialization weights from related models, accelerating fine-tuning and improving convergence stability over conventional initialization [Peebles et al., 2022].Generated initializations include weights for unseen models and LoRA parameters conditioned on example images.Source: §4.2.4 · p. 27

4.2.5. Training Acceleration

  • WSG forecasts near-future weights from historical trajectories and structural priors, allowing optimizers to skip redundant gradient steps while maintaining or improving convergence stability [Knyazev et al., 2025].WNN predicts updates with a lightweight sequence model, whereas NiNo uses neuron connectivity and graph neural networks for nowcasting.Source: §4.2.5 · pp. 27–28

4.2.6. Data Generation

  • When data is encoded as neural-network weights, WSG generates new data instances by synthesizing INR parameters for shapes, images, and 3D scenes [Dupont et al., 2022b; Chan et al., 2021].This enables model-based augmentation and generative 3D reconstruction through hypernetworks, adversarial models, VAEs, and SIREN-based generators.Source: §4.2.6 · p. 28

4.3. Discussion and Perspective

This section presents WSG as direct synthesis of neural networks in weight space, complementing WSR by modeling distributions of learners rather than only fitting functions. It highlights WSG’s capabilities, technical limitations, and three directions for more reliable, symmetry-aware generation.

Conceptual framing

  • WSG treats trained models as samples from a structured weight-space distribution, enabling neural networks to be synthesized directly rather than obtained solely through gradient-based optimization [Peebles et al., 2022, Yuan et al., 2024].WSG extends WSR’s latent model embeddings by constructing new models from that space.Source: §4.3 · p. 28

Strengths and Contributions

  • WSG exposes global model-space structure by enabling semantic model-family synthesis [Jin et al., 2024, Tian et al., 2025], solution interpolation and recombination [Navon et al., 2024, Zeng et al., 2025], and data-free model instantiation [Schürholt et al., 2022a,b].These capabilities decouple model construction from training and treat models as generatable entities.Source: §4.3 · pp. 28–29

Limitations and Open Challenges

  • WSG remains difficult because weight spaces are high-dimensional, heterogeneous, and symmetry-shaped, while generators may require many pretrained networks, be fragile to conditioning shifts [Shamsian et al., 2024, Zeng et al., 2025], and be hard to evaluate [Zhao et al., 2025b].Reliable modeling must account for architectural symmetries and assess structural and functional preservation beyond performance alone.Source: §4.3 · pp. 28–29

Outlook

  • Future work should integrate WSR invariances for geometry-aware generation, develop compact expressive latent spaces, and combine synthesis with selective fine-tuning to improve controllability and reliability.These directions aim toward composing, adapting, and creating neural networks directly in weight space.Source: §4.3 · p. 29

5. Applications to Related Domains

Weight space learning turns neural models into manipulable objects, enabling retrieval, adaptation, composition, and generation without returning to data or retraining from scratch. Across related domains, it supports architecture-agnostic workflows for signal modeling, model unification, continual and federated learning, meta-learning, and neural architecture search.

5. Applications to Related Domains

  • Weight space methods provide a unified, scalable, architecture-agnostic framework for manipulating, transferring, adapting, composing, and generating neural models directly through their parameters.This perspective connects signal modeling, model reuse, continual adaptation, distributed learning, and automated architecture design without requiring data access or training from scratch.Source: §5 · p. 29

5.1. Implicit Neural Representations

  • INRs encode continuous signals in network weights, making weight space simultaneously a structured dataset for augmentation, retrieval, transfer, classification, and signal generation.Weight-space mixup, latent INR representations, graph encoders, GANs, diffusion, and VAEs support robustness, data efficiency, storage reduction, model comparison, and synthesis of images, shapes, and scenes.Source: §5.1 · pp. 29–31

5.2. Model Unification

  • Weight-space representations unify heterogeneous architectures and modalities by embedding models into shared latent or graph-structured spaces for comparison, transfer, retrieval, clustering, and composition.GEM, Functa, Spatial Functa, uGNN, and graph-based meta-networks illustrate that shared model geometries can span fundamentally different network families.Source: §5.2 · p. 31

5.3. Continual Learning

  • Continual learning can replace mutable weights with task-conditioned regeneration, using hypernetworks and diffusion models over residual updates to recall tasks while mitigating catastrophic forgetting.This shifts memory from full models to task descriptors and supports controlled adaptation across evolving domains without overwriting prior structure.Source: §5.3 · pp. 31–32

5.4. Meta Learning

  • Meta-learning can directly synthesize task-specific weights from task descriptors, replacing costly and unstable inner-loop optimization with diffusion trajectories or hypernetwork generation.MetaDiff, Mc-Di, and Meta HyperNetworks model task-conditioned weight distributions, enabling fast, scalable, gradient-free adaptation.Source: §5.4 · p. 32

5.5. Federated Learning

  • Federated learning can communicate model-family representations rather than gradients or full updates, using server-side diffusion and hypernetworks for efficient global aggregation and personalized client models.FedDiff reduces transmission of client updates, while pFedHN and pFedGPA preserve client specialization and privacy through conditioned weight generation without accessing client data.Source: §5.5 · pp. 32–33

5.6. Neural Architecture Search

  • Weight-generating hypernetworks reframe neural architecture search as sampling weights conditioned on architecture graphs, bypassing much training-in-the-loop evaluation and enabling rapid zero-shot assessment of unseen networks.GHN predicts candidate weights in one forward pass, while GHN-1/2 scales training across diverse architectures for stronger generalization.Source: §5.6 · p. 33

6. Benchmarks

Model zoos provide the pretrained-weight collections needed to study weight space across architectures, datasets, and training regimes. The benchmarks span MLPs, CNNs, RNNs, and Transformers, supporting analysis of geometry, representations, generation, and temporal or training-regime effects.

6. Benchmarks

  • Model zoos supply structured collections of pretrained weights that enable weight-space analysis, representation learning, generative modeling, retrieval, and model-level generalization.They address the impractical cost of generating diverse model collections from scratch.Source: §6 · p. 33

6.1. Multilayer Perceptrons

  • MLP zoos provide controlled, scalable settings for studying weight geometry, similarity, alignment, interpolation, embeddings, and weight generation.Schürholt et al. [2021] construct 3.9K two-layer networks, while Implicit-Zoo spans 60K CIFAR-10, 1.4M+ ImageNet-1K, 23.4K+ Cityscapes, and 5.9K+ OmniObject3D models; Amaduzzi et al. [2025] exceed 320K NeRF models.Source: §6.1 · pp. 34–35

6.2. Convolutional Neural Networks

  • CNN zoos expose how spatial inductive biases, architectural diversity, pruning, and adapter fine-tuning shape weight-space structure across increasingly large collections.Examples include NWS [Eilertsen et al., 2020] with 16K CNNs and 320K checkpoints, Schürholt et al. [2022b] with over 3.8M weights, w2w [Dravid et al., 2024] with 60K LoRA-tuned diffusion models, and LoRA-WS with up to 100K models.Source: §6.2 · p. 35

6.3. Recurrent Neural Networks

  • RNN zoos are rarer because sequence models cost more to train, but LSTM collections capture both final solutions and temporal weight evolution.Herrmann et al. [2024] record nine checkpoints per run across 20,000 training steps for models trained on Formal Languages and Sequential MNIST.Source: §6.3 · pp. 35–36

6.4. Transformers

  • Transformer zoos support controlled study of how self-attention, normalization, residual routing, training objectives, and fine-tuning shape weight distributions and representational geometry.Falk et al. [2025a] pretrain ViT-S using seven supervised and three contrastive self-supervised seeds, then fine-tune each model under 24 CIFAR-100 configurations.Source: §6.4 · p. 36

7. Open Questions

Weight Space Learning remains early-stage, with open questions spanning first-class geometric foundations, scalable processing of large models, and robustness and safety. Progress requires architecture-agnostic representations, efficient weight-space computation, knowledge-augmented generation, and controllable defenses against adversarial risks.

7.1. Weight Space as a First-Class Learning Domain

7.2. Scaling Weight Space Learning to Large Models

  • Scaling WSL requires reducing the computational burden of full weight spaces through modular hierarchies, fine-tuning modules, and symmetry-aware compression that preserve functional properties.Proposed approaches include shared weight templates with layer-specific tokens, generation of adapters or low-rank transformations over foundation models, and approximation in symmetry-reduced low-dimensional subspaces [Khan et al., 2025, Liu et al., 2022, Ruiz et al., 2024, Shao et al., 2025, Ha et al., 2017, Wiedemer et al., 2023, Zeng et al., 2025, Jin et al., 2024, Entezari et al., 2022, Zhao et al., 2025b, Hu et al., 2022].Source: §7.2 · pp. 37–38
  • Large-model WSL is difficult because weight-space dimensionality grows with depth and width while symmetries and inter-layer dependencies complicate direct processing.Current validation is concentrated on small or medium MLPs and CNNs, while large-model efforts often use partial weight spaces or undertrained checkpoints [Ma et al., 2024, Schürholt et al., 2021, 2022b, Dravid et al., 2024, Duszenko and Bielak, 2025, Falk et al., 2025a].Source: §7.2 · p. 37

7.3. Robustness and Safety in Weight Space Operations

  • Safe WSL requires controllable generation plus detection and defense mechanisms for malicious weight edits or injected subspaces that can compromise model integrity.Future directions include task- and constraint-conditioned functionals, anomaly metrics and embedding probes, and symmetry-aware projections or reversible editing without full retraining [Ha et al., 2017, Soro et al., 2025b, Shor et al., 2025, Sun et al., 2017].Source: §7.3 · p. 38

8. Conclusion

The conclusion frames Weight Space Learning as a unified paradigm for treating neural networks as structured objects in a rich parameter space, organized around understanding, representation, and generation. It highlights WSL’s practical capabilities, evaluation resources, and growing importance for analyzing, representing, and creating models directly in weight space.

Conclusion

  • WSL unifies weight space understanding, representation, and generation into a coherent paradigm for studying neural networks as structured objects rather than only function approximators.The three dimensions cover intrinsic structure and symmetries, model-level embeddings and probing, and weight synthesis or transformation through hypernetworks and generative models.Source: §8 · p. 38
  • WSL enables practical capabilities and systematic evaluation through applications and model zoo benchmarks.These resources support evaluating and scaling research on model-centric learning.Source: §8 · p. 39
  • As pretrained models proliferate and model-centric learning grows, WSL is positioned as a foundational perspective for analyzing, representing, and creating intelligent systems in weight space.The survey calls for viewing neural networks not only as trained functions, but also as entities that can be directly analyzed, represented, and created.Source: §8 · p. 39

Related papers

Loading 2603.10090v1…