Source-linked AI summary
Deep Model Reassembly
Xingyi Yang, Daquan Zhou, Songhua Liu, Jingwen Ye, Xinchao Wang
TL;DR
Deep Model Reassembly addresses how to reuse heterogeneous pre-trained models by dissecting and recombining their building blocks under user-specified constraints. It partitions blocks into interchangeable equivalence sets and assembles customized networks using constrained optimization and a training-free performance proxy, achieving 78.6% ImageNet top-1 accuracy without fine-tuning and 83.2% with end-to-end training.
Problem
Deep Model Reassembly addresses the challenge of customizing networks from heterogeneous pre-trained models while satisfying computational and performance constraints.
Method
The method jointly partitions networks into functionally interchangeable equivalence sets, then selects blocks through constrained programming using a training-free task-performance proxy.
Results
78.6% ImageNet top-1 accuracy is achieved without fine-tuning, rising to 83.2% with end-to-end training.
Takeaways & Limitations
DeRy enables efficient customized model reuse and can aggregate knowledge from heterogeneous models while satisfying user-specified constraints.
Takeaways & Limitations
The formulation assumes a collection of N pre-trained deep neural network models, potentially trained on different tasks or with varied structures.
Abstract
from arXiv · showhide
In this paper, we explore a novel knowledge-transfer task, termed as Deep Model Reassembly (DeRy), for general-purpose model reuse. Given a collection of heterogeneous models pre-trained from distinct sources and with diverse architectures, the goal of DeRy, as its name implies, is to first dissect each model into distinctive building blocks, and then selectively reassemble the derived blocks to produce customized networks under both the hardware resource and performance constraints. Such ambitious nature of DeRy inevitably imposes significant challenges, including, in the first place, the feasibility of its solution. We strive to showcase that, through a dedicated paradigm proposed in this paper, DeRy can be made not only possibly but practically efficiently. Specifically, we conduct the partitions of all pre-trained networks jointly via a cover set optimization, and derive a number of equivalence set, within each of which the network blocks are treated as functionally equivalent and hence interchangeable. The equivalence sets learned in this way, in turn, enable picking and assembling blocks to customize networks subject to certain constraints, which is achieved via solving an integer program backed up with a training-free proxy to estimate the task performance. The reassembled models, give rise to gratifying performances with the user-specified constraints satisfied. We demonstrate that on ImageNet, the best reassemble model achieves 78.6% top-1 accuracy without fine-tuning, which could be further elevated to 83.2% with end-to-end training. Our code is available at https://github.com/Adamdad/DeRy
1 Introduction
Deep Model Reassembly (DeRy) reuses heterogeneous pretrained models by dissecting and selectively recombining their building blocks under user-specified constraints. A two-stage constrained optimization strategy makes this feasible and achieves strong ImageNet transfer performance.
- Motivation and task: DeRy introduces a knowledge-transfer task that customizes networks by reassembling building blocks from heterogeneous pretrained models under computational constraints.It aims to aggregate knowledge without increasing computation cost while preserving or enhancing downstream performance.
- Motivation and task: The task is challenging because model architectures are unconstrained and reassembled models need lightweight, training-free performance estimates.These requirements concern both extracting compatible blocks and selecting effective assemblies without retraining.
- Method: DeRy first partitions pretrained networks into functionally equivalent, interchangeable block sets, then selectively assembles blocks into customized models.The two optimization stages are decoupled after the equivalence sets are learned.
- Method: The partition stage uses cover set optimization, while reassembly searches block combinations with a training-free proxy based on counting linear regions in feature representations.The proxy reduces the cost of searching candidate reassemblies.
- Contributions: The paper contributes a constrained-programming framework for customized model reuse and reports competitive transfer performance, sometimes surpassing model-zoo candidates.The authors connect these results to knowledge aggregation across pretrained models and universal connectivity among neural networks.
2 Related Work
Prior model-zoo transfer methods select one model, ensemble models, or fuse models with identical or homogeneous structures. DeRy instead rearranges blocks from heterogeneous pretrained networks.
- Model-zoo transfer: Model-zoo transfer commonly selects a single model through fine-tuning or transferability estimation, but unreliable estimates can yield suboptimal choices.The related-work discussion identifies model selection as one of three major solution families.
- Model-zoo transfer: Ensemble methods combine multiple pretrained models but incur prohibitive computational costs at test time.
- Model-zoo transfer: Adaptive fusion methods reduce models to a single target model but generally require identical or homogeneous network structures.This limits their applicability to diverse architectures in typical model zoos.
- Deep Model Reassembly: DeRy differs by dissecting pretrained models into building blocks and rearranging them to create new pretrained models.Its framing extends transfer from reusing whole models toward block-level recombination.
3 Deep Model Reassembly
Deep Model Reassembly formulates model reuse as partitioning heterogeneous networks into functionally equivalent blocks and reassembling them under performance and resource constraints. The approach uses constrained optimization and a training-free proxy to make block selection tractable without full retraining.
- Problem Formulation: Deep Model Reassembly seeks an L-layer model maximizing task performance subject to hard computational or parametric constraints.
- Network Partition by Functional Equivalence: The method partitions networks into depth-wise blocks and groups functionally similar blocks into equivalence sets for interchangeability.Functional equivalence is defined through similarity of block outputs for similar input batches.
- Network Partition by Functional Equivalence: The partitioning objective seeks disjoint, approximately size-balanced sub-graphs with maximum internal functional similarity.Blocks are assigned to equivalence sets using an assignment matrix and representative anchor blocks.
- Network Partition by Functional Equivalence: KL-style layer swapping and K-Means-style clustering optimize partitions and equivalence-set assignments, with multiple random runs used to select the best result.The implementation repeats optimization for R = 200 seeds and retains the best partition.
- Network Reassembly by Solving an Integer Program: A data-dependent linear-region count estimates transfer performance without training, reducing the cost of evaluating candidate reassemblies.The proxy is motivated by network expressivity and the difficulty of distinguishing inputs with similar binary codes.
4 Experiments
Experiments evaluate DeRy’s partitioning, interchangeable blocks, training-free proxy, ImageNet performance, transfer learning, and ablations. Results indicate that the approach supports effective reassembly across constraints and tasks.
- Experimental Setup: The model zoo combines manually designed CNNs, NAS-based architectures, and attention-based Vision Transformer and Swin-Transformer models.Weights are collected from Torchvision, timm, and OpenMMLab.
- Partition Properties: Equivalence sets tend to cluster blocks by stage index, suggesting similar patterns are learned at similar network stages.The observation includes bottom layers from varied pre-trained networks appearing in the same equivalence set.
- Training-Free Proxy: NASWOT provides a reasonable training-free predictor of transferability, showing high Kendall’s Tau correlation with fine-tuned accuracy.The proxy is evaluated on 10 pre-trained models across 8 image-classification tasks and on the timm model zoo for ImageNet.
- ImageNet Evaluation: 78.6% Top-1 accuracy is achieved by DeRy(4,90,20) with 1.27M trainable parameters under ImageNet evaluation.DeRy(4,10,3) reaches 76.9% after 100 epochs and 78.4% after 300 epochs.
- Transfer Learning: DeRy produces diverse structures, from lightweight MobileNetv3-based models to large CNN–Swin hybrids, while improving short-training accuracy over Swin-T by 0.9% and 0.2% in reported comparisons.The comparisons use less parameter and computational requirements than the Swin-T model.
- Transfer Learning: DeRy generally surpasses models trained from scratch under matched computational constraints and exceeds pre-trained candidates on Cars, Aircraft, and Flower.With ImageNet pre-training, DeRy can outperform the best candidate in the original model zoo, with gains increasing as parameter constraints are relaxed.
5 Conclusion
DeRy provides a practical framework for reassembling heterogeneous pretrained networks into customized models under user-defined constraints. It uses jointly learned equivalence sets and a training-free proxy to support efficient block selection and achieves strong transfer-learning performance.
- Both the network partitioning and block-selection stages are crucial to the overall approach.
- DeRy partitions heterogeneous pretrained networks into equivalence sets whose blocks are treated as functionally interchangeable.The partitions are learned jointly through cover set optimization.
- The method selects and assembles blocks by solving an integer program with a training-free task-performance proxy.This enables customized networks without requiring retraining during the search.
- DeRy achieves gratifying performance across a series of transfer-learning benchmarks while satisfying user-defined constraints.