PaperlayerView exact revision on arXiv

Source-linked AI summary

Network Morphism

Tao Wei, Changhu Wang, Yong Rui, Chang Wen Chen

arXiv:1603.01670v2cs.LGcs.CVcs.NE

TL;DR

The paper addresses how to expand trained neural networks without changing their function, avoiding the long retraining associated with exploring new architectures. It develops equations and algorithms for diverse morphing operations and nonlinear activations, then reports benchmark effectiveness and a VGG transformation using much less training time.

  • Problem

    Training effective deep networks can take weeks or months, while existing knowledge-transfer approaches may alter the network function; the paper seeks function-preserving network transformation.

  • Method

    The paper derives network morphism equations and algorithms for depth, width, kernel-size, and subnet changes in classic and convolutional networks, using parametric-activation families for continuous nonlinearities.

  • Results

    The scheme is effective on benchmark datasets and morphs a 16-layer VGG network to a better-performing model with 1/15 of scratch-training time.

  • Takeaways & Limitations

    Network morphism provides a function-preserving route for expanding trained networks while retaining inherited knowledge and enabling continued growth in shortened training time.

  • Takeaways & Limitations

    IdMorph cannot handle commonly used non-idempotent activations such as Sigmoid and TanH; the proposed morphism investigation is limited to expanding architectures.

Abstract

from arXiv · show

We present in this paper a systematic study on how to morph a well-trained neural network to a new one so that its network function can be completely preserved. We define this as \emph{network morphism} in this research. After morphing a parent network, the child network is expected to inherit the knowledge from its parent network and also has the potential to continue growing into a more powerful one with much shortened training time. The first requirement for this network morphism is its ability to handle diverse morphing types of networks, including changes of depth, width, kernel size, and even subnet. To meet this requirement, we first introduce the network morphism equations, and then develop novel morphing algorithms for all these morphing types for both classic and convolutional neural networks. The second requirement for this network morphism is its ability to deal with non-linearity in a network. We propose a family of parametric-activation functions to facilitate the morphing of any continuous non-linear activation neurons. Experimental results on benchmark datasets and typical neural networks demonstrate the effectiveness of the proposed network morphism scheme.

1. Introduction

Network morphism aims to transform a trained parent network into a deeper or wider child while preserving its function and inherited knowledge. The paper addresses diverse architectural changes, nonlinear activations, and shortened retraining through new equations and algorithms.

  • Training effective deep networks can take weeks or months, motivating methods that reuse well-trained networks with minimal retraining.
  • Network morphism transfers parameters from a parent to a child while preserving the parent’s function and outputs.
  • The proposed framework supports depth, width, kernel-size, and subnet morphing for classic multilayer perceptrons and convolutional networks.
  • A deconvolution-based depth-morphing algorithm can asymptotically fill parameters with non-zero elements and exceeds IdMorph’s worst-case occupancy by an order of magnitude.
  • Parametric-activation function families reduce nonlinear operations to learnable linear forms, enabling morphism for continuous nonlinear activations.
  • Experiments report effectiveness on benchmark datasets and morph a 16-layer VGG network to a better-performing model using 1/15 of scratch-training time.

2. Related Work

Earlier approaches transfer knowledge through imitation, pre-training, or transfer learning, whereas NetMorph preserves the network function while supporting broader morphing operations and nonlinear activations.

  • Mimic Learning: Mimic-learning methods train student networks to reproduce teacher outputs, commonly requiring training from scratch.
  • Pre-training and Transfer Learning: Pre-training and transfer learning facilitate convergence or adaptation, but reinitialize final layers and alter the network function.
  • Net2Net: Net2Net is restricted to identity-layer morphing, idempotent activations, and depth or width changes.
  • Net2Net: NetMorph handles arbitrary nonlinear activations and depth, width, kernel-size, and subnet changes, including simultaneous depth, width, and kernel-size morphing.

3. Network Morphism

The paper organizes network morphism around basic operations whose combinations can produce complex transformations while maintaining unchanged outputs.

  • Depth, width, kernel-size, and subnet morphing are presented as basic operations that can be combined into complex network morphisms.

3.1. Network Morphism: Linear Case

The linear-case construction inserts an intermediate layer or convolutional block whose composed transformation equals the parent transformation. A unified morphism equation covers fully connected and convolutional networks while exposing width and kernel-size choices.

  • The linear analysis first removes nonlinear activations and models classic networks as sequences of fully connected layers.
  • Depth morphing inserts a hidden layer Bl between parent layers Bl−1 and Bl+1 while requiring Bl+1 = Fl+1 · (Fl · Bl−1) = G · Bl−1.
  • For classic networks, morphism becomes a matrix decomposition problem in which G is factored into Fl and Fl+1.
  • In DCNNs, hidden layers are 3D blobs and weights are 4D convolutional filters whose shapes encode channels and kernel sizes.
  • The convolutional construction requires Fl, Fl+1, and G to satisfy a convolutional composition equation, using a zero-padded G with effective kernel size K1 + K2 − 1 ≥ K.
  • Equation (6) is the network morphism equation, and its choices of Cl, K1, and K2 simultaneously determine depth, width, and kernel-size changes.

3.2. Network Morphism Algorithms: Linear Case

The linear-case algorithms solve network morphism equations by iteratively decomposing expanded filters, balancing parameter density and scale while preserving the parent operation. A practical variant guarantees exact solutions in expanding settings and improves non-zero parameter occupancy over IdMorph.

  • Algorithm Design: The algorithms favor parameters with many non-zero elements and consistent scale, reflecting initialization and convergence considerations.The general and practical variants differ in whether dense parameter filling is guaranteed under a condition or achieved asymptotically without that condition.
  • General Network Morphism: General network morphism initializes child kernels with random noise and alternately solves each kernel by deconvolution while fixing the other.The loss decreases during iterations, although Algorithm 1 is not universally guaranteed to converge to zero.
  • General Network Morphism: Algorithm 1 converges to zero loss when either child kernel has at least as many parameters as the expanded target filter.The condition is framed as having more unknowns than constraints in an undetermined linear system.
  • Practical Network Morphism: Algorithm 2 repeatedly applies Algorithm 1 while shrinking an auxiliary kernel size until the loss reaches zero.The procedure terminates because the loss is guaranteed to be zero when the auxiliary kernel size reaches one.
  • Practical Network Morphism: NetMorph achieves O(C^2) non-zero elements in the worst case and O(C^2K^2) in the best case, versus IdMorph’s O(C).Under C ≫ K, NetMorph can asymptotically fill all parameters with non-zero elements, while sacrificing the non-sparse guarantee in the practical algorithm.

3.3. Network Morphism: Non-linear Case

The non-linear case addresses the failure of identity-based morphing beyond idempotent activations by introducing continuously learnable activation families. Setting the inserted activation linear initially preserves the function, after which its non-linearity can be learned.

  • Challenge: Replacing one activated layer with two activated layers is generally non-trivial because the added non-linearity can change the network function.The difficulty applies to arbitrary activation functions in the proposed two-layer replacement.
  • IdMorph Limitation: IdMorph works for ReLU but not Sigmoid or TanH because its identity-layer construction requires an idempotent activation function.The required condition is ϕ◦ϕ = ϕ.
  • P-Activation Functions: A P-activation family continuously maps an activation ϕ to the linear identity transform, enabling morphing for arbitrary continuous non-linear activations.The canonical family is defined by ϕ_a = (1 − a) · ϕ + a · ϕ_id for a ∈ [0,1].
  • Morphing Procedure: During morphing, the inserted activation is initialized with a = 1 so it is linear, then becomes non-linear as a is learned during continued training.The surrounding green activations can be safely added under the illustrated construction.

3.4. Stand-alone Width and Kernel Size Morphing

Stand-alone width and kernel-size morphing expand convolutional structures while preserving the parent computation. Width expansion uses complementary channels and controlled sparsity; kernel expansion pads filters and blobs with matching zeros.

  • Width Morphing: Width morphing expands layer Bl from Cl channels to ˜Cl channels, with ˜Cl ≥ Cl, while keeping adjacent parent layers specified for function preservation.The construction tracks original channels and their complement in the expanded child blob.
  • Width Morphing: For complementary channels, one adjacent filter block is set to zero while the other receives arbitrary values, and a random channel permutation breaks clustering.The smaller-parameter block is zeroed to follow the non-sparse practice.
  • Kernel Size Morphing: Kernel-size morphing pads each filter and its corresponding blobs with (˜Kl − Kl)/2 zeros on every side, preserving their values and operation.The resulting filters and blobs have the same shape and values.

3.5. Subnet Morphing

Subnet morphing expands a single parent layer into sequential or stacked subnet structures. The procedure first solves sequential expansion and then applies it across split paths for stacked designs.

  • Motivation: The motivating subnet examples are the mlpconv layer in NiN and the inception layer in GoogLeNet.These templates support constructing deeper networks from repeated subnet structures.
  • Subnet Morphing: Subnet morphing transforms a minimal number of parent layers, typically one, into a child subnet such as a stacked sequential subnet.The inception layer in GoogLeNet is described as a four-way stacking of sequential subnets.
  • Sequential Subnet Morphing: Sequential subnet morphing changes one layer into P + 1 sequential layers using a network morphism equation derived for the expanded structure.The equation can be solved by iteratively optimizing one layer’s parameters while fixing the others.
  • Sequential Subnet Morphing: The effective kernel size of the sequential construction is determined from the component kernel sizes after zero-padding.The paper denotes the resulting effective size as ˜K.
  • Stacked Sequential Subnet Morphing: Stacked sequential subnet morphing first splits one parent layer into multiple paths, then performs sequential subnet morphing on each path.The illustrated workflow includes an n-way stacked construction with one path morphed into two layers.

4. Experimental Results

Experiments on MNIST, CIFAR10, and ImageNet evaluate network morphism across classic and convolutional networks, multiple morphing operations, and activation functions. NetMorph generally improves accuracy over parent models, Net2Net, and scratch training while substantially reducing training time in the ImageNet experiment.

  • Experimental scope: Experiments cover MNIST, CIFAR10, and ImageNet, testing depth, width, kernel-size, and subnet morphing on classic and convolutional networks.The evaluations include ReLU and TanH-related activation settings.
  • Classic neural networks: On MNIST, NetMorph improves accuracy from 92% to 97%, while Net2Net improves only to 94%.NetMorph also handles PTanH, for which Net2Net is unavailable.
  • Depth and subnet morphing: On CIFAR10 depth morphing, NetMorph improves performance from 78.15% to 82.06% and then 82.43%, compared with Net2Net reaching 81.21% and 81.99%.The comparison follows the morphing sequence cifar_111→211→222→2222→3333.
  • Depth and subnet morphing: Subnet morphing raises CIFAR10 performance from 82.43% to 83.14%, followed by further improvement to around 84%.These results correspond to morphing cifar_222 to cifar_2222 and then cifar_2222 to cifar_3333.
  • Internal regularization: NetMorph consistently outperforms scratch-trained Raw models on CIFAR10, with the gap becoming larger as the network gets deeper.The paper interprets this pattern as internal regularization from learning parameters in multiple phases.
  • Kernel size and width morphing: Kernel-size morphing reaches 82.81% from an 81.48% parent, while width morphing reaches 83.09% for NetMorph versus 82.70% for Net2Net.The kernel expands the second-layer kernels from 1 to 3, and width doubles first-layer channels.
  • ImageNet: For ImageNet VGG16, NetMorph outperforms its parent, the multi-scale version, and VGG19, while completing the 19-layer morphing and training process within 5 days.The reported training time represents around 15x speedup over training from scratch.

5. Conclusions

The paper introduces a systematic network morphism scheme that preserves a parent network’s function while expanding its architecture. It supports diverse morphing operations and continuous non-linear activations, with experiments demonstrating effectiveness.

  • Conclusion: Network morphism transforms a well-trained parent network into a child network while completely preserving the network function.The child can inherit the parent’s knowledge and grow into a more powerful network in a short time.
  • Conclusion: The scheme supports diverse morphing operations and algorithms based on derived morphism equations.Supported changes include depth, width, kernel size, and subnet structure.
  • Conclusion: The proposed algorithms enable morphing with any continuous non-linear activation neurons.The paper reports extensive experiments demonstrating the scheme’s effectiveness.
Loading 1603.01670v2…