PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Model Reassembly

Xingyi Yang, Daquan Zhou, Songhua Liu, Jingwen Ye, Xinchao Wang

arXiv:2210.17409v2cs.CVcs.AI

TL;DR

Reusing pretrained networks as whole models leaves open how to dissect heterogeneous model collections into blocks and reassemble customized networks under constraints. DeRy partitions networks into equivalence sets and assembles blocks using optimization and a training-free performance proxy. Its reassembled model exceeds 78% top-1 accuracy on ImageNet with all blocks frozen.

  • Problem

    Reusing pretrained models largely as whole networks leaves the challenge of reassembling heterogeneous models into customized networks, with feasibility and retraining-free performance estimation unclear.

    Source: §1 · pp. 1–2

  • Method

    DeRy jointly partitions pretrained networks into functionally interchangeable equivalence sets, then uses constrained optimization and a training-free performance proxy to assemble customized models.

    Source: §1 · p. 2, §5 · p. 10

  • Results

    More than 78% top-1 accuracy on ImageNet was achieved by a reassembled model with all blocks frozen; fine-tuning further elevated performance, sometimes surpassing any model in the zoo.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The approach is presented as applicable to varied backbone families, including CNNs, transformers, and MLPs.

    Source: §1 · p. 2

    The formulation assumes access to a zoo of pretrained deep neural networks, whose models may be trained on different tasks or have varied structures.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

In this paper, we explore a novel knowledge-transfer task, termed as Deep Model Reassembly (DeRy), for general-purpose model reuse. Given a collection of heterogeneous models pre-trained from distinct sources and with diverse architectures, the goal of DeRy, as its name implies, is to first dissect each model into distinctive building blocks, and then selectively reassemble the derived blocks to produce customized networks under both the hardware resource and performance constraints. Such ambitious nature of DeRy inevitably imposes significant challenges, including, in the first place, the feasibility of its solution. We strive to showcase that, through a dedicated paradigm proposed in this paper, DeRy can be made not only possibly but practically efficiently. Specifically, we conduct the partitions of all pre-trained networks jointly via a cover set optimization, and derive a number of equivalence set, within each of which the network blocks are treated as functionally equivalent and hence interchangeable. The equivalence sets learned in this way, in turn, enable picking and assembling blocks to customize networks subject to certain constraints, which is achieved via solving an integer program backed up with a training-free proxy to estimate the task performance. The reassembled models, give rise to gratifying performances with the user-specified constraints satisfied. We demonstrate that on ImageNet, the best reassemble model achieves 78.6% top-1 accuracy without fine-tuning, which could be further elevated to 83.2% with end-to-end training. Our code is available at https://github.com/Adamdad/DeRy

1 Introduction

Deep Model Reassembly (DeRy) reuses heterogeneous pre-trained networks by dissecting them into blocks and reassembling customized models under user-specified constraints. Its two-stage optimization groups interchangeable blocks into equivalence sets, then selects a reassembly using a training-free performance estimate.

1 Introduction

  • Unlike reuse of pre-trained models as wholes, DeRy aims to aggregate knowledge across heterogeneous models to customize networks without increasing computation cost.The paper presents this as a new knowledge-transfer task for producing networks tailored to user-specified constraints.Source: §1 · pp. 1–2
  • DeRy addresses the feasibility challenge of extracting blocks from architecture-unconstrained model zoos and estimating reassembled-model performance without retraining.Reassembled models are also expected to satisfy parametric constraints and behave reasonably well.Source: §1 · p. 2
  • DeRy partitions heterogeneous networks into functionally interchangeable equivalence sets, then selectively reassembles blocks to meet user specifications through a two-stage optimization strategy.The partitioning and reassembly steps are decoupled, so fixed equivalence sets can support later customizations.Source: §1 · pp. 1–2
  • A cover set optimization derives the equivalence sets, while linear-region counts estimate transferability during training-free reassembly search, reducing search cost by 10^4 versus exhaustive training.The approach uses constrained programming and avoids retraining candidate reassemblies for the performance estimate.Source: §1 · p. 2
  • >78% ImageNet top-1 accuracy is achieved by reassembled models with all blocks frozen; finetuning can further improve performance and sometimes surpass any model in the zoo.DeRy can also accommodate diverse backbones, including CNNs, transformers, and MLPs.Source: §1 · p. 2

2 Related Work

The related work distinguishes DeRy from model-zoo transfer by its block-level reassembly of diverse pre-trained networks, and connects its approach to neural representation similarity and model stitching.

Transfer learning from Model Zoo

  • Model-zoo transfer selects one model, ensembles models, or fuses models, but fusion assumes identical or homogeneous structures; DeRy instead rearranges blocks from diverse models.Standard transfer fine-tunes one model [75] [42] [32] [86] [30] or imparts knowledge to other models [28] [76] [61] [77] [43]. Model-zoo approaches include model selection [36] [65] [75], transferability scoring [51] [66] [64] [4], ensembles [1], and fusion of identical [62] [68] or homogeneous [63] [52] structures; transferability estimates may be unreliable, and ensembles costly at test time.Source: §2 · pp. 2–3

Neural Representation Similarity

  • DeRy uses representation similarity to quantify function-level distance between neural blocks, drawing on methods including CCA, CKA, and generalized shape metrics.The passage also lists linear regression [28], CCA [57], CKA [37], and generalized shape metrics [70] as representation-similarity methods.Source: §2 · p. 3

Network Stitching

  • Unlike bottom-to-top stitching of network layers, including ResNet-to-visual-transformer stitching, DeRy investigates assembling arbitrary pre-trained network structures.Model stitching was initially proposed in [41] and is also described in [2] [13]; a recent work reports competitive performance stitching a visual transformer atop ResNet [65].Source: §2 · p. 3

3 Deep Model Reassembly

DeRy partitions heterogeneous pretrained networks into functionally similar blocks, then reassembles blocks into task-specific models under computational or parameter constraints. It uses a training-free proxy to estimate candidate performance.

3.1 Problem Formulation

  • The model zoo may contain networks trained on different tasks or with varied structures; DeRy seeks the best-performing L-layer composition for a task subject to hard computational or parameter constraints.A 1×1 convolution stitching layer adjusts feature dimensions between consecutive blocks when they mismatch.Source: §3.1 · pp. 3–4
  • On CIFAR-100, a reassembled ResNet50 improved accuracy over its iNaturalist2021 and ImageNet-1k predecessors by 0.63% and 2.73%, respectively.The reported assembly combines the bottom two stages of the iNaturalist2021 ResNet50 with stages 3 and 4 of the ImageNet-1k ResNet50.Source: Table 3 · p. 4, §3.1 · p. 4

3.2 Network Partition by Functional Equivalence

  • Blocks are treated as interchangeable when they have high internal functional similarity, defined by similar outputs for similar input batches.The simplified similarity measure sums input and output similarity; the partition objective also favors blocks of approximately similar size. A KL-style partition search and alternating K-means-style clustering are repeated across 200 seeds.Source: §3.2 · p. 5

3.3 Network Reassembly by Solving an Integer Program

  • DeRy selects one block from each equivalence set and stage to maximize target-task performance under parameter and FLOPs limits via a 0-1 integer program.Rather than train each candidate, it estimates transfer performance using data-dependent linear regions; random-search candidates are ranked by the NASWOT log(det K) score computed from feature-vector Hamming distances.Source: §3.3 · p. 6

4 Experiments

The experiments examine DeRy’s reassembly properties and evaluate its transfer performance under computational constraints. They report ImageNet and downstream-task results, along with analyses of block similarity, proxy scoring, and search ablations.

4 Experiments

  • The model zoo contains 21 architectures and 30 pretrained weight sets from Torchvision, timm, and OpenMMLab, spanning CNN, NAS-based, and attention-based models.Experiments use K=4, epsilon=0.2, 500 candidates per constraint setting, and frozen-tuning or full-tuning protocols.Source: §4 · pp. 6–7

4.1 Exploring the Properties for Deep Reassembly

  • Same-stage replacement blocks yield at least 70% top-1 accuracy, similarity ranks performance within a target network, and learned equivalence sets cluster blocks by stage.Linear CKA features also show a diagonal pattern, with representations most similar at the same stage.Source: Figure 4 · p. 7, Figure 5 · p. 7, §4.1 · p. 7
  • Training a DeRy architecture from scratch causes a substantial performance drop relative to using both reassembled structures and pretrained weights, indicating contributions from both factors.Source: §4.1 · p. 7
  • NASWOT provides a reasonable predictor of transferability, with high Kendall’s Tau correlation against fine-tuned accuracy across 10 pretrained models on 8 tasks and the timm zoo on ImageNet.The reported evaluations compare NASWOT scores with test accuracy.Source: §4.1 · p. 7, Figure 6 · p. 8

4.2 Transfer learning with Reassembled Model

  • DeRy(4,90,20) reaches 78.6% top-1 accuracy with 1.27M trainable parameters; DeRy(4,10,3) reaches 76.9% after 100 epochs and 78.4% after 300.The 100-epoch result is reported as surpassing all small-sized models in the model zoo.Source: Table 7 · p. 8, §4.2 · p. 8
  • DeRy models improve accuracy over Swin-T by 0.9% and 0.2% under SHORT-TRAINING with fewer parameters and computation, spanning MobileNetv3-based designs to large CNN-Swin hybrids.The reported structures include DeRy(4,10,3), which adopts lightweight blocks, and DeRy(4,90,20), which forms a large CNN-Swin hybrid.Source: Table 9 · p. 8, Table 10 · p. 8, Figure 8 · p. 8
  • DeRy generally surpasses scratch-trained models under the same computational constraints, beats pretrained candidates on Cars, Aircraft, and Flower, and DeRy+In1k can exceed the model zoo’s best candidate.The reported performance improvement grows as parameter constraints increase; LogME and LEEP do not necessarily select the optimal model.Source: Figure 11 · p. 9, §4.2 · p. 9
  • The training-free proxy reduces reassembly search cost by 10^4 with marginal performance degradation, while cover-set partitioning improves transfer performance and reduces the reassembly search space.The ablation reports 100-epoch frozen-tuning top-1 accuracy and search time under DeRy(4,50,10); similarity computation is excluded because it is performed offline.Source: Table 2 · p. 9, §4.2 · pp. 9–10

5 Conclusion

Deep Model Reassembly (DeRy) is a knowledge-transfer task that reassembles building blocks from heterogeneous pre-trained networks into models subject to user-defined constraints. The proof-of-concept demonstrates practical efficiency and reports strong transfer-learning performance while offering insight into functional similarity between networks.

5 Conclusion

  • DeRy achieves gratifying performance on transfer-learning benchmarks and reveals functional similarity between neural networks through stitching heterogeneous models.The conclusion describes these results as outcomes of the proof-of-concept solution.Source: §5 · p. 10
  • DeRy partitions pre-trained networks jointly with cover set optimization, then uses equivalence sets and an integer program with a training-free performance proxy to select and assemble blocks.The assembled models are customized to satisfy user-defined constraints.Source: §5 · p. 10
  • DeRy deconstructs heterogeneous pre-trained neural networks into building blocks and reassembles them into models subject to user-defined constraints.The paper presents a proof-of-concept intended to show that this task is possible and practically efficient.Source: §5 · p. 10

Related papers

Loading 2210.17409v2…