Source-linked AI summary
Elastoformer: Enabling Dynamic Adaptivity via Elastic Model Transformation
Sudaksh Kalra, Dolly Sapra
TL;DR
EdgeAI vision systems must operate under changing latency, power, and memory constraints that fixed-computation networks cannot efficiently accommodate. Elastoformer transforms a pretrained network into a single elastic architecture with multiple shared descendant modes selected at runtime. It achieves broad efficiency gains while retaining competitive performance across supported architectures, with compression performance depending on the core model capacity.
Problem
EdgeAI deployments face fluctuating latency, power, memory, and computation constraints, whereas conventional networks use fixed computational workflows.
Method
Elastoformer uses a two-step Compress and Grow framework with weight sharing to create multiple Descendant Networks from one pretrained model for runtime selection.
Results
Elastoformer reduces FLOPs by up to 85%, memory overhead by up to 76%, and inference latency by up to 50% while retaining more than 90% of peak original performance.
Takeaways & Limitations
The framework provides architecture-agnostic elastic inference across Vision Transformers and CNNs without managing multiple independent models.
Takeaways & Limitations
At higher compression ratios, Descendant Networks may fail to reach high performance; lower compression can recover oracle performance with 33% of original FLOPs.
Abstract
from arXiv · showhide
EdgeAI systems are increasingly employing computer vision applications to enable intelligent, on-device decision-making in real-time. However, these deployments face highly dynamic operational conditions, with fluctuating constraints on latency, power availability, and memory resources. Deep Neural Networks (DNN), which follow fixed computational execution flows, lack the flexibility to adapt to such variability, resulting in inefficient and suboptimal performance in edge scenarios. This underscores the need for architectures that are not only efficient but also dynamically scalable at runtime. In this paper, we propose Elastoformer: A framework that transforms conventional neural networks (NN) into Elastic NN capable of real-time elastic inference. Unlike the conventional bag-of-models approach, which requires maintaining multiple independent models for different operating conditions, Elastoformer offers a single, modular solution that dynamically switches between multiple modes of operation at runtime, adapting efficiently to the changing computational budgets of edge devices without the overhead of managing separate models. Experiments reveal that our framework achieves up to 85% reduction in computation FLOPs, 50% reduction in latency and 76% reduction in memory overhead, while showcasing the architecture agnostic nature of the framework across both Vision Transformers and CNNs. Our code is available at https://github.com/sudaksh14/Elastoformer.
1 Introduction
EdgeAI deployments face changing compute, memory, and energy constraints, while conventional networks follow fixed execution flows. Elastoformer transforms pretrained models into elastic architectures with multiple runtime configurations from one shared model.
- Dynamic EdgeAI resource availability requires models that adapt computational demands to current system conditions.
- Elastoformer transforms existing pretrained models into elastic architectures that adapt to evolving EdgeAI conditions while maintaining competitive performance.
- A two-step Compress and Grow framework creates multiple Descendant Networks from one pretrained model for different operating modes and compression levels.
- The framework dynamically adapts model size and FLOPs, applies across vision transformers and CNNs, and avoids maintaining separate independent models.
- Weight sharing reuses parameters across Descendant Networks, reducing memory overhead and transition latency while limiting retraining costs.
2 Background and Motivation
EdgeAI applications operate under fluctuating resource budgets and task demands, making fixed neural networks inefficient or risky across operating contexts. Elastoformer embeds multiple scalable configurations in one architecture to adjust inference cost at runtime with low switching overhead.
- EdgeAI systems must balance latency, energy, memory, and computation as resource availability and operating conditions change.
- Fixed architectures waste energy in low-demand settings and can create safety-critical risks when lightweight models face demanding conditions.
- The drone scenario illustrates fluctuating computational and energy requirements, but it is presented only as a motivational example rather than a specific commercial platform.
- Elasticity enables runtime changes to depth, width, or computational paths in response to latency, energy, memory, or input-complexity conditions.
- Elastoformer integrates multiple execution modes into one shared architecture, enabling computational scaling with minimal switching overhead under changing constraints.
3 Related Works
Prior efficiency and dynamic-network methods reduce inference cost or support multiple configurations, but each has architectural, compression, or deployment limitations. These limitations motivate broader elastic approaches for resource-constrained edge systems.
- Pruning removes redundant parameters to create smaller networks with limited performance loss, using unstructured or structured strategies.
- Slimmable Neural Networks support multiple width configurations through shared weights and switchable batch normalization, but depend on backbones using batch normalization.
- AdaptiveNet provides runtime elasticity through a NAS-initialized supernet, but its approximately 3× overparameterization creates substantial overhead.
- LegoDNN enables block-grained runtime scaling, but limits compression by leaving input- and output-connected portions uncompressed.
- Early-exit networks reduce computation through confidence-based intermediate predictions, while adding overhead for confidence evaluation.
4 Elastoformer Methodology
Elastoformer transforms a pre-trained network into a family of shared-weight Descendant Networks through Compress and Grow, enabling runtime selection across resource-accuracy trade-offs. The method combines structured pruning, metadata-based weight restoration, and partial freezing to support elastic inference with minimal duplication.
- Framework Overview: Elastoformer uses Input Specification, Elastic Transformation, and Deployment and Runtime Selection to convert a pre-trained model into runtime-selectable Descendant Networks.Elastic Transformation applies the two-step Compress and Grow pipeline.
- Neural Network Compression: Structured pruning progressively removes parameters from attention and feed-forward layers while reducing hidden dimensions in LayerNorm and Patch Embedding layers.The approach also extends pruning to CNN convolutional weights through binary masks.
- Neural Network Compression: L1-norm saliency identifies lower-importance attention rows for masking, avoiding costly second-order gradient methods.Rows with the lowest average absolute magnitude are masked while the remaining rows are retained.
- Neural Network Compression: The desired Compression Ratio is determined from the original and minimum FLOPs under the edge system's latency constraints.The formulation also estimates the pruning ratio per round.
- Neural Network Growth: Grow reverses pruning incrementally, restoring stored weights to construct larger Descendant Networks from the smallest Core Network.Pruning metadata records the original positions and values needed for restoration.
- Neural Network Growth: Weight sharing reuses inherited weights across Descendant Networks, while partial gradient freezing updates only newly added parameters.New networks combine preserved weights with reactivated weights, supporting warm-start initialization and avoiding weight duplication.
- Elastic Neural Network: The resulting Elastic Neural Network is a collection of Descendant Networks spanning distinct compression levels and resource-accuracy trade-offs.The Core Network is the smallest configuration, while successive networks add parameters and computational cost through progressive growth.
5 Experimental Setup
Experiments evaluate Elastoformer on ImageNet, CIFAR-10, and CIFAR-100 using ViT-B and additional CNN backbones. Evaluation measures accuracy-resource trade-offs, latency, memory requirements, switching memory, and training cost against recent efficient and adaptive methods.
- Datasets: The evaluation uses ImageNet, CIFAR-10, and CIFAR-100 benchmark vision datasets.CIFAR-10 and CIFAR-100 each contain 50,000 training and 10,000 testing images, with 10 and 100 classes respectively.
- Backbones and Baselines: ViT-B is the pre-trained transformer backbone, and comparisons include efficient vision transformers, Early-Exit ViT, and an elastic slimmable approach.ViT-B has 12 layers, embedding dimension 768, and patch size 16.
- Backbones and Baselines: Additional experiments on ResNet-50 and VGG-16 assess whether the approach generalizes beyond transformer architectures.CNN comparisons use recent methods for early-exit and adaptive dynamic neural networks.
- Metrics: Evaluation measures Top-1 accuracy against descendant-network parameters and FLOPs, alongside latency and memory requirements.The study also assesses switching memory and GPU hours when applying the framework to a new pre-trained network.
- Training Procedure: Each Descendant Network is trained for 50 epochs with five epochs of linear warmup using eight NVIDIA GeForce RTX 3090 GPUs.Training uses PyTorch, AdamW, cosine annealing, and repeated augmentation.
- Training Procedure: Selective gradient clipping is applied only to parameters not shared with previous Descendant Networks.This implementation choice is consistent with the method's partial weight freezing mechanism.
6 Evaluation and Results
Elastoformer provides elastic inference across ViTs and CNNs, enabling accuracy–resource trade-offs through multiple descendant networks while reducing computation, latency, and memory requirements. Its evaluations show runtime efficiency gains, but design time scales with the number of descendant networks and dataset size, and core-model capacity constrains performance at high compression.
- Performance vs Resource Trade-off: 85% FLOPs reduction at CR = 0.6 retained more than 90% of peak original performance, while CR = 0.4 achieved up to 66% reduction with accuracy loss ≤2%.These results demonstrate a spectrum of accuracy–efficiency trade-offs across compression ratios.
- Performance vs Resource Trade-off: Elastoformer provided a better resource trade-off than Matformer under similar resource constraints.
- CNN Generality: The elastification process extended to ResNet-50 and VGG-16, preserving original-network performance and reducing FLOPs by up to 75%.On CIFAR datasets, the transformed networks also achieved improved accuracy.
- Latency: 50% latency reduction across both Jetson platforms was achieved with less than a 2% performance drop.The latency evaluation used generated descendant networks on Jetson Orin and Jetson Nano.
- Runtime Adaptation: Switching between descendant networks took approximately 50.4 ms, corresponding to about one inference cycle and limiting effective downtime to a single cycle.The authors report negligible impact on sustained throughput during frequent runtime adaptations.
- Memory Efficiency: Memory savings reached 76% versus AdaptiveNet for ResNet-50 and 60% versus a bag-of-models approach for ViT.Weight sharing enabled multiple operating modes with a single parameter set, with approximately 1 MB metadata overhead per descendant network.
- Design Time: ImageNet design time scaled approximately as 10^N hours with N descendant networks, while CIFAR-100 design time was typically ≤10% of ImageNet’s.The design-time evaluation used multiple descendant-network counts and datasets.
- Limitations and Future Work: At CR = 0.8, descendant networks could not reach high performance, whereas at CR = 0.4 the core model regained oracle performance using 33% of original FLOPs.The authors identify core-model capacity as a major determinant of overall system performance.
7 Conclusion
Elastoformer transforms static neural architectures into elastic networks that adapt to dynamic resource constraints in real-time edge environments. It generates descendant networks from one pre-trained model, reducing resource demands while preserving predictive performance and extending across transformer and convolutional architectures.
- Elastoformer transforms static Vision Transformers into elastic neural networks for real-time adaptation to dynamic edge-resource constraints.
- Multiple descendant networks generated from one pre-trained model provide a flexible trade-off between computational cost and predictive performance.
- 76% memory overhead reduction results from Elastoformer’s weight-sharing mechanism compared with conventional methods.
- 50% lower inference latency is achieved relative to the original Vision Transformer.
- Elastoformer reduces FLOPs by up to 85% while retaining more than 90% of peak original performance.
- The framework generalizes to ResNet-50 and VGG-16, demonstrating applicability beyond Vision Transformers.