Source-linked AI summary
On the Interaction Between Model Compression and Test-Time Adaptation
Francesco Corti, Dong Wang, Young D. Kwon, Cecilia Mascolo, Olga Saukh
TL;DR
The paper asks whether structured compression preserves models’ ability to adapt to distribution shifts, a key issue for efficient deployment. It evaluates compression and TTA across architectures and datasets using diagnostics of representation diversity and gradient compatibility, finding that TTA degrades as compression increases while supervised adaptation remains comparatively effective. The results indicate method-dependent losses in adaptability and motivate compression strategies that explicitly preserve it.
Problem
The interaction between model compression and test-time adaptation is poorly understood, especially whether structured compression preserves adaptability under distribution shift.
Method
The study evaluates structured compression with backpropagation-based TTA across ResNet-18 and ViT-Base using diagnostics for representational expressivity, diversity, and adaptation-gradient compatibility.
Results
TTA performance progressively falls behind supervised adaptation as compression increases, with failures involving representational collapse and gradient degeneracy or divergence.
Takeaways & Limitations
Compression should be selected and budgeted for adaptability rather than source accuracy alone, with the preferred strategy depending on the compression method and TTA objective.
Takeaways & Limitations
The study covers two architectures and limited TTA methods, and generalization to larger models remains future work.
Abstract
from arXiv · showhide
Deep neural networks deployed in the wild must be both efficient and adaptable, requiring model compression and test-time adaptation (TTA). While both are well studied in isolation, their interaction remains poorly understood. We systematically analyze how structured compression affects a model's ability to adapt under distribution shift. Using ResNet-18 and ViT-Base on CIFAR-10-C and ImageNet-C, we evaluate multiple compression methods combined with standard TTA techniques. We introduce a diagnostic framework that examines representational expressivity and adaptation subspace compatibility. Our results reveal a consistent gap: although compressed models retain high accuracy under supervised adaptation, their TTA performance degrades significantly with increasing compression. We show that this stems from reduced representational diversity and structural constraints that limit recoverability. These effects strongly depend on the compression method, highlighting the need to design compression strategies that preserve adaptability.
1 INTRODUCTION
This section frames the underexplored interaction between structured compression and test-time adaptation in resource-constrained deployment. The study finds that increasing compression widens the TTA–supervised adaptation gap through representational and gradient-alignment failures, with effects varying by method and dataset.
- Motivation: Compression and adaptation address simultaneous deployment constraints, but their interaction and impact on adaptability under distribution shift remain poorly understood.The paper asks whether structured compression preserves a model’s ability to adapt at test time.
- Motivation: Edge adaptation is memory-intensive, making compression a prerequisite for some on-device adaptation scenarios.Entropy-minimization adaptation can trigger out-of-memory errors on resource-constrained devices even at small batch sizes.
- Empirical findings: TTA performance degrades as compression increases while supervised adaptation remains strong, widening the gap across architectures and datasets, especially on ImageNet-C.Figure 1 averages across corruption types and compares post-compression TTA with supervised fine-tuning.
- Diagnostic framework: Compression causes gradient degeneracy and active divergence, so entropy- and consistency-based TTA signals can collapse or oppose supervised update directions.The supervised cross-entropy gradient can remain more informative at the same sparsity.
- Empirical findings: The severity depends on compression method and dataset: Wanda and Taylor preserve more recoverable structure, Folding is more stable, and effects are stronger on ImageNet-C than CIFAR-10-C.OBD and Mag-ℓ2 show pronounced degradation.
2 RELATED WORK
Prior work studies structured compression, test-time adaptation, and loss-landscape behavior largely as separate topics. The remaining gap is identifying which structural properties compression must preserve for effective adaptation.
- Structured Model Compression: Structured pruning removes entire computational units, while folding clusters similar channels into shared representations for efficient inference.Selection criteria include magnitude, activation-aware, Taylor, and Hessian-based scores.
- Test-Time Adaptation: TTA adapts models to distribution shifts using unlabeled data through methods such as entropy minimization, reliable filtering, regularization, and consistency objectives.The literature includes methods designed for convolutional networks and vision transformers.
- Loss Landscape Geometry and Mode Connectivity: Loss-landscape research links pruning recovery and plasticity to connectivity and stability properties in parameter space.Pruned subnetworks recover full accuracy only when stable to SGD noise, according to the cited prior work.
- Open question: The central unresolved question is which structural properties a compression method must preserve to enable effective adaptation.This motivates examining representational and geometrical properties under structured compression.
3 DIAGNOSTIC FRAMEWORK
The diagnostic framework evaluates compression–TTA interaction through representational expressivity and adaptation-subspace compatibility. Across architectures and shifts, structured compression limits recoverability and can make unsupervised TTA gradients degenerate or oppose supervised updates.
- 3.1 EXPRESSIVITY AND DIVERSITY: The framework combines CKA-based representational alignment with Activation Map Entropy to measure recovery of expressivity and diversity after compression.CKA compares compressed representations with the dense model, while AME measures entropy of activation Gram-matrix spectra.
- 3.1 EXPRESSIVITY AND DIVERSITY: AME recovery is constrained by an entropy ceiling under higher sparsity, while ViT-Base is less sensitive because FFN-only compression preserves post-residual dimensionality.On CIFAR-10-C, saturation appears at 35% and 55% sparsity; on ImageNet-C, Oracle achieves tighter and lower-distance recovery than TTA.
- 3.2 SUBSPACE COMPATIBILITY: Structured compression reduces the admissible adaptation subspace and can misalign its remaining update directions with the target-domain loss.This motivates testing whether compression reduces the fraction of useful adaptation signal accessible to TTA.
- 3.2 SUBSPACE COMPATIBILITY: Compression creates gradient degeneracy or active divergence: TTA gradients may vanish near-uniform predictions or remain large while opposing supervised gradients.Entropy and consistency objectives can collapse under compression, whereas supervised cross-entropy remains informative when prediction confidence drops.
4 DISCUSSION, OUTLOOK AND LIMITATIONS
The study finds that structured compression can preserve source or supervised-adaptation accuracy while silently reducing unsupervised test-time adaptability. The authors recommend adaptability-aware compression and joint design of compression budgets with TTA objectives, while identifying important scope boundaries for future work.
- Discussion: Compressed models can retain source-domain accuracy yet progressively lose test-time adaptability under unsupervised objectives.The paper describes this as silent plasticity loss and reports degeneration of entropy-based and consistency-based TTA gradients under compression.
- Practical guidelines: Compression criteria should be selected for adaptability, because Wanda and Taylor preserve higher representational recoverability and gradient alignment than Mag-ℓ2 and OBD, while Folding performs best among data-free methods.These comparisons are reported in Figures 2 and 4.
- Practical guidelines: Compression budgets and TTA objectives should be co-designed: pruning alignment reverses sign between 45% and 65% sparsity on CIFAR-10-C, while objective sensitivity differs on ImageNet-C.For Folding, the sign reversal occurs near 75% sparsity; entropy objectives diverge earlier than consistency objectives on ImageNet-C.
- Outlook: Future work includes compression-aware TTA objectives and broader evaluation of unstructured pruning, quantization, combined compression, and backpropagation-free adaptation.The authors also note that their diagnostics are mainly gradient-based despite reporting additional backpropagation-free accuracy curves.
- Limitations: The study’s scope is limited to ResNet-18 and ViT-Base with two TTA methods per architecture, so generalization to larger models remains open.Experiments focus primarily on maximum corruption severity 5, with additional results at severity 3 and across multiple seeds.
A APPENDIX
The appendix evaluates how compression affects adaptation gaps, representation similarity, gradient alignment, computational cost, and recovery across architectures, corruptions, and TTA methods. Across these analyses, increasing compression generally widens the Oracle–TTA gap and worsens adaptation-related behavior, with effects varying by compression method.
- Adaptation gap: The per-corruption Oracle–TTA adaptation gap generally widens as compression increases across data-dependent pruning and data-free compression criteria.The analysis covers Wanda, Taylor, OBD, Folding, and Mag-ℓ2 across individual corruption types.
- Representation similarity: Pre-adaptation worst-layer CKA is positively correlated with post-adaptation accuracy, strongly for ViT-Base/ImageNet-C and weakly for ResNet-18.Bars report Spearman correlations at severity 3 and severity 5 across corruptions.
- Evaluation robustness: Multi-seed analyses compare SAR with matched Oracle-SAR baselines using identical optimization settings but different loss definitions.Shaded regions represent ±σ across three random seeds of corruption-averaged accuracy.
- Evaluation robustness: The compression-induced Oracle–TTA gap and its widening trend persist across different pre-training recipes, indicating robustness to the particular pre-training run.The multi-seed checkpoints shift absolute post-adaptation accuracy relative to the single-checkpoint analysis.
- Additional diagnostics: The appendix also reports corruption-family-specific gradient alignment, geometry-aware similarity, compute and memory profiles, and TTA behavior under increasing compression.Additional figures cover Noise, Blur, Weather, and Digital corruptions; CPU latency, adaptation time, FLOPs, memory, SAR, Oracle-SAR, PEA, and FOA.
- Additional diagnostics: Removing confident-but-wrong samples from SAR’s reliability-filtered set restores gradient alignment to approximately +0.9 across corruptions.The figure compares the original cosine-similarity curves with Oracle-SAR evaluated on correctly classified samples.
- Additional diagnostics: With increasing compression, TTA and Oracle collapse for pruned models, while smaller dense models with matched parameter budgets provide the reference comparison.At low sparsity, compressed-model TTA remains competitive with the smaller-dense references.