PaperlayerView exact revision on arXiv

Source-linked AI summary

NeRN -- Learning Neural Representations for Neural Networks

Maor Ashkenazi, Zohar Rimon, Ron Vainshtein, Shir Levi, Elad Richardson, Pinchas Mintz, Eran Treister

arXiv:2212.13554v2cs.LGcs.CV

TL;DR

Learning a compact implicit representation of a pretrained CNN is nontrivial. NeRN maps architecture-position coordinates to CNN kernel weights and combines smoothness and distillation-inspired losses; the paper reports satisfying reconstruction results on popular architectures and benchmarks and presents weight-importance analysis and meta-compression applications.

  • Problem

    A larger predictor can overfit a smaller original network, but successfully learning a compact implicit representation is nontrivial.

    Source: §1 · p. 1

  • Method

    NeRN trains a predictor MLP to map each convolutional kernel's architecture-position coordinate to its pretrained weights, using reconstruction, distillation, and smoothness methods.

    Source: §1 · pp. 1–2, §3.2 · p. 3, §3.3 · p. 4

  • Results

    NeRN yields satisfying reconstruction results on popular architectures and benchmarks, and the paper demonstrates the importance of weight smoothness and two applications.

    Source: §6 · p. 9

  • Takeaways & Limitations

    NeRN can support weight-importance analysis through its accuracy and meta-compression by pruning the predictor, possibly without task data.

    Source: §6 · p. 9

    The regularization-based smoothness approach requires modifying original-network training and access to its training data, and may degrade accuracy.

    Source: §3.3 · p. 4

Abstract

from arXiv · show

Neural Representations have recently been shown to effectively reconstruct a wide range of signals from 3D meshes and shapes to images and videos. We show that, when adapted correctly, neural representations can be used to directly represent the weights of a pre-trained convolutional neural network, resulting in a Neural Representation for Neural Networks (NeRN). Inspired by coordinate inputs of previous neural representation methods, we assign a coordinate to each convolutional kernel in our network based on its position in the architecture, and optimize a predictor network to map coordinates to their corresponding weights. Similarly to the spatial smoothness of visual scenes, we show that incorporating a smoothness constraint over the original network's weights aids NeRN towards a better reconstruction. In addition, since slight perturbations in pre-trained model weights can result in a considerable accuracy loss, we employ techniques from the field of knowledge distillation to stabilize the learning process. We demonstrate the effectiveness of NeRN in reconstructing widely used architectures on CIFAR-10, CIFAR-100, and ImageNet. Finally, we present two applications using NeRN, demonstrating the capabilities of the learned representations.

1 INTRODUCTION

NeRN represents a pretrained CNN by mapping architecture-position coordinates to its kernel weights. The paper addresses the difficulty of learning a compact implicit representation with weight smoothness methods and a distillation-inspired training scheme.

1 INTRODUCTION

  • NeRN maps each (layer, filter, channel) coordinate to the corresponding pretrained CNN kernel weights, then reconstructs the network by querying all possible coordinates.Each coordinate corresponds to the weights of a k × k convolutional kernel.Source: §1 · p. 1
  • A larger predictor can overfit a smaller original network, but learning a compact implicit representation is nontrivial.The authors propose adding smoothness over the original network weights to address this challenge.Source: §1 · p. 1
  • Weight smoothness can be promoted with regularization during original-network training or with post-training permutations of its weights.The paper also proposes a distillation-inspired training scheme for better, more stable optimization.Source: §1 · p. 2
  • A trained NeRN represents the specific network it was trained on, and its reconstruction error can be used to study weight importance.The paper reports reconstruction experiments on classification benchmarks and provides a scalable PyTorch framework.Source: §1 · p. 2

2 RELATED WORK

Related work uses coordinate-driven neural representations for varied signals and uses predictor networks to generate neural-network weights. NeRN adopts an MLP for convolutional weights and draws on offline response- and feature-based distillation.

2 RELATED WORK

  • Neural representations use coordinate inputs to an MLP to represent signals; prior work demonstrates them for 3D shapes, images, video, and other tasks.The cited work reports superiority of implicit 3D shape representations over grids or meshes [1, 2, 3, 4, 5, 6, 7].Source: §2 · p. 2
  • Prior neural representations adapt predictor architectures to their signals, while this work uses a simple MLP with additional methods for convolutional weights.Chen et al. (2021) adopted a CNN for frame prediction.Source: §2 · p. 2
  • Weight prediction generates network weights with an additional predictor; unlike task-trained HyperNetworks [1], this work represents a pretrained network using fixed inputs.Other cited approaches use model datasets to predict hyperparameters or weight initialization, or use graph inputs for unseen architectures [1, 2, 3].Source: §2 · p. 2
  • Knowledge distillation can use response logits or feature maps and can be offline or online; this work uses offline response- and feature-based distillation.The cited examples for response-based methods include Ba & Caruana (2014) and Hinton et al. (2014); feature-based methods include Romero et al. (2015).Source: §2 · p. 2

3 METHOD

NeRN represents a pretrained convolutional network by mapping kernel coordinates to weights, then trains the predictor with weight, output-logit, and feature-map losses. It promotes learnable weight smoothness through regularization or post-training permutations.

3.1 DESIGNING NERNS

  • Each (layer, filter, channel) coordinate maps to a convolutional kernel through positional embeddings and a five-layer MLP; fully connected and normalization layers are omitted.The output size is set to the largest kernel size, with smaller kernels sampled from its middle.Source: §3.1 · p. 3

3.2 TRAINING NERNS

  • NeRN combines weight reconstruction, logit knowledge distillation, and feature-map distillation; these losses improve accuracy, convergence, and stability over reconstruction alone without direct task loss.The authors state that training requires no labeled data and may not require data at all.Source: §3.2 · p. 3, Figure 1 · p. 3

3.3 PROMOTING SMOOTHNESS

  • Training-time smoothness regularization encourages similarity between kernels, but requires changing the original training scheme and access to training data, and may reduce task accuracy.The approach adds a smoothness loss while training the original network.Source: §3.3 · p. 4
  • Permutation-based smoothness reorders the kernels NeRN predicts without changing the original weights, approximating a minimum-weight Hamiltonian path with a greedy nearest-unvisited-kernel search.The path is defined over a graph whose vertices are kernels and whose edge weights are cosine distances, or L2 distances for 1 × 1 kernels.Source: §3.3 · pp. 4–5
  • Cross-filter permutations incur 4%-6% storage overhead for standard ResNet variants, versus 2%-3% for in-filter permutations.The greedy algorithm does not guarantee that cross-filter permutations are superior.Source: §3.3 · p. 5

4 EXPERIMENTS

NeRN reconstruction is evaluated on ResNet vision benchmarks, with experiments probing predictor size, weight smoothness, data-free inputs, non-smooth weights, and loss and sampling ablations.

4 EXPERIMENTS

  • Experiments evaluate ResNet architectures on CIFAR-10, CIFAR-100, and ImageNet, examining predictor sizes and weight smoothness alongside data-free, non-smooth, and ablation settings.CIFAR-10 results are complete; CIFAR-100 and ImageNet show only the best setup in the main text, with complete results in the appendix.Source: §4 · p. 5

4.1 CIFAR-10

  • CIFAR-10 reconstruction improves with predictor size; regularization trades source accuracy against reconstruction, with optimal factors 5 · 10−6 (size 240) and 1 · 10−6 (size 280).Figure 3 shows the regularization-factor comparisons for hidden sizes 240 and 280.Source: Table 1 · p. 6, §4.1 · pp. 6–7, Figure 3 · p. 6

4.2 CIFAR-100

  • On CIFAR-100, promoting smoothness and using larger predictors improve reconstruction; the task requires a slightly larger NeRN than CIFAR-10 for the same reconstructed architecture.The reported results use ResNet56 and about 90k NeRN training iterations.Source: §4.2 · p. 7, Table 2 · p. 7

4.3 IMAGENET

  • For ImageNet ResNet18, a NeRN about 54% of the original model’s size gives satisfying results while predicting only 3 × 3 convolutions, over 98% of convolutional parameters.The setup skips the first 7 × 7 convolution and downsampling layers and uses post-training permutations for smoothness.Source: §4.3 · p. 7, Table 3 · p. 7

4.4 DATA-FREE TRAINING

  • With combined losses, NeRN achieves good reconstruction from uniformly sampled noise X ∼ U[−1, 1], without meaningful input data from the original task.Table 4 compares noise-input results with omitting distillation losses under cross-filter permutations.Source: §4.4 · pp. 7–8, Table 4 · p. 8

4.5 RECONSTRUCTING NON-SMOOTH NETWORKS

  • NeRN can reconstruct non-smooth networks, but results are inferior to smooth-network reconstruction; the accuracy gap is more significant for smaller predictors.Smaller predictors typically have less capacity to capture non-smooth signals.Source: §4.5 · p. 8, Table 5 · p. 8

4.6 ABLATION EXPERIMENTS

  • Loss ablations show distillation losses improve reconstruction and enable faster, more stable convergence, while a reconstruction loss is still needed; a separate ablation examines weight sampling.Figure 4 shows the convergence comparison; Table 7 examines sampling methods for gradient computation.Source: §4.6 · p. 8, Figure 4 · p. 8, Table 6 · p. 8

5 ADDITIONAL APPLICATIONS

NeRN supports weight-importance analysis and meta-compression, with a further proposed use for representing already-pruned networks.

5 ADDITIONAL APPLICATIONS

  • NeRN’s lower-reconstruction-error filters correspond to high-valued activation maps, suggesting a way to identify important filters.The authors hypothesize that distillation losses lead NeRN to prioritize weights by their influence on activations and logits.Source: §5 · p. 9
  • Naive magnitude-based pruning of the NeRN predictor is proposed to make its representation more disk-size economical.The predictor is itself a neural network; more sophisticated options such as structured pruning and quantization are suggested for future exploration.Source: §5 · p. 9
  • Representing an already-pruned network with NeRN is proposed as a further disk-size reduction direction, potentially without task data.The authors reason that fewer weights simplify the prediction task and likely reduce NeRN’s size.Source: §5 · p. 9

6 CONCLUSION

The conclusion presents NeRN as a predictor MLP for reconstructing pretrained CNN weights, highlights weight smoothness and two applications, and describes reproducibility materials.

6 CONCLUSION

  • NeRN uses a predictor MLP to reconstruct the weights of a pretrained CNN.Source: §6 · p. 9
  • The paper reports satisfying reconstruction results on popular architectures and benchmarks and emphasizes promoting smoothness in the original network’s weights.It presents weight-importance analysis and predictor pruning for disk-size compactness as possible applications.Source: §6 · p. 9
  • The supplementary materials include a generic scalable NeRN framework, configuration files, and a README with reproduction instructions.Source: §6 · p. 10

Appendices

The appendices show a qualitative kernel reconstruction example and describe the initialization used for NeRN.

A VISUALIZING RECONSTRUCTED KERNELS

B NERN INITIALIZATION

  • NeRN is initialized so its predicted weights initially have means and variances similar to those of the original model’s weights.The authors report that this HyperNetworks-like initialization empirically allows relatively faster convergence.Source: B NERN INITIALIZATION · p. 13

C CONVOLUTIONAL PARAMETERS SIZE

The appendix table concerns convolutional parameter sizes in standard ResNet architectures.

C CONVOLUTIONAL PARAMETERS SIZE

  • The table covers convolutional parameter sizes in standard ResNet architectures.Source: Table 8 · p. 13

D.1 SIZE OVERHEAD

This section quantifies permutation-storage overhead and illustrates how cross-filter permutations rearrange kernels across filters.

D.1 SIZE OVERHEAD

  • Cross-filter permutations require F_l · C_l · (log2 F_l + log2 C_l) bits to save, versus F_l · C_l · log2 C_l + F_l · log2 F_l bits for in-filter permutations.Table 9 reports size overhead for both variants on the paper's standard ResNet architectures.Source: §D.1 · p. 14, Table 9 · p. 14

D.2 PERMUTATIONS ILLUSTRATION

  • Cross-filter permutation concatenates a layer’s kernels, permutes the combined tensor, then reshapes it into filters, so kernels may leave their original filters.In the illustration, kernel colors indicate their original filter membership, not their weight values.Source: Figure 7 · p. 14, §D.2 · p. 14

E.1 CIFAR10 WITH REGULARIZATION-BASED SMOOTHNESS

The appendix provides reconstruction-result tables for CIFAR-10, CIFAR-100, and ImageNet, and reports experiments on positional embeddings that did not support the proposed inductive bias.

E.1 CIFAR10 WITH REGULARIZATION-BASED SMOOTHNESS

  • Table 10 reports CIFAR-10 reconstruction results for ResNet56 with regularization-based smoothness and 3.25 MB of learnable weights.The caption notes a tradeoff between original-network accuracy and NeRN’s ability to reconstruct the network.Source: Table 10 · p. 15

E.2 CIFAR100

  • Table 11 presents complete NeRN reconstruction results on CIFAR-100.Source: Table 11 · p. 15

E.3 IMAGENET

F A NOTE ON POSITIONAL EMBEDDINGS

  • Experiments refuted the hypothesis that positional embeddings designed for adjacent similarity and distant separability aid reconstruction of smooth networks.Figure 8 compares normalized cosine similarity to coordinate (0, 0, 31) across channel coordinates and positional-embedding base frequencies; further investigation is left for future research.Source: F A NOTE ON POSITIONAL EMBEDDINGS · p. 16, Figure 8 · p. 16

Related papers

Loading 2212.13554v2…