PaperlayerView exact revision on arXiv

Source-linked AI summary

EcoTTA: Memory-Efficient Continual Test-time Adaptation via Self-distilled Regularization

Junha Song, Jungsoo Lee, In So Kweon, Sungha Choi

arXiv:2303.01904v4cs.CV

TL;DR

Continual test-time adaptation can be memory-intensive and can suffer from error accumulation and catastrophic forgetting during long-term adaptation. EcoTTA adapts frozen original networks with lightweight meta networks and self-distilled regularization, improving memory efficiency and maintaining stable adaptation; it uses 86% and 80% less memory than CoTTA for ResNet-50 and WideResNet-40, respectively.

  • Problem

    Existing test-time adaptation methods under-explore memory reduction despite on-device constraints, while long-term adaptation with noisy unsupervised loss can cause error accumulation and catastrophic forgetting.

  • Method

    EcoTTA freezes the original network, adapts lightweight meta networks, and regularizes their outputs toward those of the frozen network using self-distillation without extra memory.

  • Results

    EcoTTA improves memory efficiency and TTA performance across image classification and semantic segmentation benchmarks, using 86% and 80% less memory than CoTTA for ResNet-50 and WideResNet-40, respectively.

  • Takeaways & Limitations

    Self-distilled regularization keeps performance consistent during long-term adaptation, while the architecture supports continual TTA on memory-limited edge devices.

  • Takeaways & Limitations

    EcoTTA requires source data to warm up the newly attached meta networks for a small number of epochs before deployment.

Abstract

from arXiv · show

This paper presents a simple yet effective approach that improves continual test-time adaptation (TTA) in a memory-efficient manner. TTA may primarily be conducted on edge devices with limited memory, so reducing memory is crucial but has been overlooked in previous TTA studies. In addition, long-term adaptation often leads to catastrophic forgetting and error accumulation, which hinders applying TTA in real-world deployments. Our approach consists of two components to address these issues. First, we present lightweight meta networks that can adapt the frozen original networks to the target domain. This novel architecture minimizes memory consumption by decreasing the size of intermediate activations required for backpropagation. Second, our novel self-distilled regularization controls the output of the meta networks not to deviate significantly from the output of the frozen original networks, thereby preserving well-trained knowledge from the source domain. Without additional memory, this regularization prevents error accumulation and catastrophic forgetting, resulting in stable performance even in long-term test-time adaptation. We demonstrate that our simple yet effective strategy outperforms other state-of-the-art methods on various benchmarks for image classification and semantic segmentation tasks. Notably, our proposed method with ResNet-50 and WideResNet-40 takes 86% and 80% less memory than the recent state-of-the-art method, CoTTA.

1. Introduction

Test-time adaptation addresses domain shift but faces practical memory constraints and instability during long-term, changing-target adaptation. EcoTTA combines lightweight meta networks with self-distilled regularization to reduce memory use while preserving source knowledge and stabilizing adaptation.

  • Test-time adaptation adapts models to unlabeled online target data when source and target domains differ.
  • Activation storage, rather than learnable parameters, is the primary training-memory bottleneck, making parameter-only updates insufficient for constrained devices.Activations are intermediate features stored during forward propagation for gradient calculations during backpropagation.
  • Changing target domains create long-term risks of catastrophic forgetting and error accumulation from noisy unsupervised losses.Catastrophic forgetting degrades source-domain performance, while unavailable target labels force reliance on noisy adaptation signals.
  • EcoTTA freezes the original network and adapts lightweight meta networks attached to it, reducing the intermediate activations needed for backpropagation.The meta networks comprise one batch-normalization and one convolution block.
  • Self-distilled regularization keeps meta-network outputs close to frozen-network outputs, preserving source knowledge while preventing error accumulation and catastrophic forgetting without extra memory.The regularization runs in parallel with the adaptation loss and adds negligible overhead.
  • EcoTTA improves memory efficiency and adaptation performance across image-classification and semantic-segmentation benchmarks.The reported tasks include CIFAR10/100-C, ImageNet-C, and Cityscapes with weather corruption.

2. Related Work

Related work addresses domain shift through adaptation, continual adaptation, and memory-efficient on-device learning. EcoTTA combines these concerns by reducing activation memory while targeting continual TTA under changing domains.

  • Mitigating domain shift: Domain generalization seeks invariant representations, whereas test-time adaptation directly adapts models to test data to address domain shift.
  • Test-time adaptation: Most TTA methods improve unsupervised losses or small-batch usability while focusing on stationary target domains.
  • Test-time adaptation: Continual TTA addresses continuously changing targets but can accumulate errors and suffer catastrophic forgetting during long-term exposure to unlabeled data.
  • Efficient on-device learning: On-device learning shows that activation size, rather than learnable parameter count, bottlenecks training memory on constrained devices.Prior TTA work largely focused on reducing learnable parameters instead.
  • Efficient on-device learning: EcoTTA reduces the high activation sizes required for TTA while also improving adaptation performance.

3. Approach

EcoTTA reduces continual TTA memory by adapting lightweight meta networks attached to frozen, partitioned original networks. Self-distilled regularization constrains meta-network outputs using frozen-network outputs to preserve source knowledge and limit long-term adaptation failures.

  • Memory-efficient Architecture: Freezing original layers avoids storing their intermediate activations, leaving only the lightweight meta networks to be updated during adaptation.Frozen-layer backpropagation requires weights but not saved activations, whereas learnable layers require intermediate activations for weight gradients.
  • Memory-efficient Architecture: EcoTTA partitions the pretrained encoder into K parts and attaches a meta network to each part before deployment.The meta networks are pretrained on source data, while the original network remains frozen during target-domain adaptation.
  • Adaptation Objective: Entropy minimization adapts the meta networks using unlabeled target data, with an indicator selecting samples whose prediction entropy is below H0.The main adaptation loss uses the prediction output ˆy and softmax probabilities p(·).
  • Self-distilled Regularization: Self-distilled regularization uses an L1 loss to keep each meta-network output ˜xk close to the corresponding frozen-network output xk.The regularization is added to the adaptation objective to address catastrophic forgetting and error accumulation.
  • Self-distilled Regularization: The regularizer preserves source-domain knowledge and class discriminability without storing additional original networks.The frozen-network outputs provide the distilled reference, so the method adds no extra memory for a separate teacher model.

4. Classification Experiments

Classification experiments evaluate continual TTA across CIFAR and ImageNet corruption benchmarks, showing that EcoTTA combines competitive adaptation with lower memory use. Ablations further examine architecture, partitioning, and regularization for stable long-term adaptation.

  • Benchmarks: EcoTTA is evaluated on CIFAR10-C, CIFAR100-C, and ImageNet-C, using sequential corruption adaptation without resetting the model.The target domains contain 15 corruption types with five severity levels.
  • Comparisons: EcoTTA achieves the lowest memory usage while outperforming existing TTA methods across the evaluated CIFAR pre-trained models.With WideResNet-40, it uses 80% less memory than CoTTA and 58% less than EATA while achieving superior performance.
  • Comparisons: On ImageNet-C, EcoTTA matches CoTTA performance while using 86% less memory with K=4 and 75% less with K=5.It performs better when the ResNet-50 backbone uses strong data augmentation such as AugMix.
  • Architecture ablations: Dense partitioning of shallow encoder blocks improves TTA more than dense partitioning of deep blocks.The reported comparison favors a 2,2,4,4 shallow-to-deep allocation over 4,4,2,2.
  • Partitioning: K=4 or 5 gives the best TTA performance, whereas larger K values increase activations and learnable parameters without reducing average error further.Small K values barely modify intermediate outputs, limiting adaptation performance.
  • Regularization: Self-distilled regularization keeps performance consistent during long-term adaptation, while removing it causes overfitting and error accumulation.The regularization study varies λ and evaluates repeated corruption sequences to test stability.

5. Segmentation Experiments

EcoTTA is tested on continual semantic segmentation under repeated weather corruptions. It achieves the highest mIoU while substantially reducing memory and avoiding the instability observed with continual TENT.

  • Experimental setup: The segmentation benchmark repeats four Cityscapes-C weather corruptions ten times to simulate continual domain shifts.Experiments use a ResNet-50-based DeepLabV3+ model with batch size 2.
  • Results: EcoTTA achieves the highest mIoU while using 66% less memory than competing continual TTA.The result is reported for Cityscapes continual TTA with DeepLabV3Plus-ResNet-50.
  • Results: Unlike continual TENT, EcoTTA avoids catastrophic forgetting and error accumulation during the repeated-corruption evaluation.The reported outcome combines stable long-term adaptation with the highest mIoU.

6. Conclusion

The paper concludes that EcoTTA improves continual TTA performance while reducing memory requirements. Its frozen-network architecture and self-distilled regularization target memory efficiency, source-knowledge preservation, and long-term stability.

  • Conclusion: EcoTTA is designed for continual TTA on edge devices with limited memory.The conclusion frames memory savings and improved continual adaptation as the method’s central outcomes.
  • Conclusion: The architecture reduces memory by decreasing intermediate activations used for gradient calculations.It combines frozen original networks with trainable meta networks.
  • Conclusion: Self-distilled regularization constrains meta-network outputs toward the original networks to preserve source knowledge and prevent error accumulation.The conclusion also links the regularizer to preventing catastrophic forgetting during long-term adaptation.
  • Conclusion: Experiments on diverse datasets and backbone networks verify EcoTTA’s memory efficiency and TTA performance.The conclusion presents these evaluations as evidence supporting the approach.

A. Efficiency for TTA methods

The efficiency analysis explains why adapting only lightweight meta networks reduces stored activations during backpropagation. Runtime comparisons indicate minor overhead relative to CoTTA, while memory savings remain substantial.

  • Memory efficiency: Updating full model parameters requires additional memory to store activations, whereas frozen layers can avoid saving activations unnecessary for backward propagation.The analysis distinguishes activation storage requirements across learnable and frozen layers.
  • Memory efficiency: Figure 6 depicts forward and backward propagation through activations and linear layers to clarify the memory-saving design.Black lines denote forward propagation and red lines denote backward propagation.
  • Computational efficiency: Wall-clock time is used to compare computational cost because common FLOPs libraries generally support inference rather than training.The evaluation includes the practical cost of test-time adaptation.
  • Computational efficiency: PyTorch limitations prevent wall-clock measurements from fully representing the computational efficiency of EATA and EcoTTA.EATA’s filtering is not reflected in fine-grained gradient computation, and EcoTTA’s implementation similarly requires theoretical-time reporting.
  • Results: EcoTTA requires negligible overhead compared with CoTTA while using 80% less memory than CoTTA and 58% less than EATA.Measurements include loading data and adapting to all 15 corruptions on one NVIDIA 2080Ti GPU.

B. Discussion and further experiments

The experiments examine continual and gradually changing TTA, memory usage, regularization, and semantic segmentation. EcoTTA outperforms prior methods while reducing memory and limiting long-term error accumulation.

  • Gradually changing setup: EcoTTA outperforms previous TTA methods under a gradually changing corruption-severity setup.The sequence changes corruption severity across 15 corruption types on CIFAR100-C.
  • Parameter-efficient transfer learning: 177% more memory and a 6.1% performance drop occur when comparing PETL+SDR with EcoTTA.The comparison uses WRN-40 on CIFAR100-C.
  • Regularization comparison: Self-distilled regularization achieves superior performance to alternative regularizations in the controlled comparison.The architecture and adaptation loss remain fixed while only the regularizer is replaced.
  • Method implications: The method uses frozen original networks and self-distilled regularization without requiring a large activation size or an extra source model.The approach can be combined with different adaptation losses, including TENT and EATA.
  • Semantic segmentation: EcoTTA achieves comparable segmentation performance with 85% less memory than CoTTA.Its performance remains consistent across multiple rounds, whereas continual TENT suffers error accumulation.

C. Further implementation details

Implementation details describe how EcoTTA partitions pretrained networks, configures meta-network convolutions, and warms up meta networks. The design emphasizes denser partitioning in shallow layers and architecture-specific convolution choices.

  • Partition of a pre-trained model: EcoTTA divides the encoder into K parts, with denser partitions in shallow layers to improve TTA performance.For ResNet, the encoder comprises layers 1–4, and the method is applied to that encoder.
  • Convolution layer in meta networks: A kernel size of three improves WideResNet performance with 10% additional model parameters.For ResNet, the same kernel size instead causes significant parameter and memory increases.
  • Warm-up transformations: The same simple transformations warm up meta networks across pretrained architectures and methods, including AugMix-pretrained WideResNet-40.The transformations combine ColorJitter, GaussianBlur, and RandomGrayscale.
  • Semantic segmentation setup: Semantic segmentation experiments warm up meta networks on Cityscapes for five epochs using SGD with learning rate 5e-2.The experiments use ResNet-50-based DeepLabV3+.
  • Meta-network components: The meta-network ablation evaluates affine transformation, standardization, convolution-block components, and input choices.The table reports which components are applied to the meta networks.

D. Additional ablations

Additional ablations compare adaptation losses, meta-network components, regularization losses, and regularization weights. The results identify a preferred adaptation loss while showing robustness to regularization choices.

  • Main task loss for adaptation: Using L3, which filters samples with entropy minimization, achieves the lowest error rate in most cases.Across the three tested losses, all methods achieve performance comparable to EATA, and L3 is selected for EcoTTA.
  • Components of meta networks: Removing affine transformation causes a larger performance drop than removing standardization in the batch-normalization layer.Using only a convolution layer also degrades performance, making the ReLU and BN layers jointly important in the convolution block.
  • Regularization loss: EcoTTA is robust to the choice of regularization loss, with only a marginal performance difference between the two tested losses.The comparison evaluates the regularization functions while retaining the proposed architecture and adaptation setting.
  • Regularization weight: A change in λ from 0.5 to 1 changes first-round performance by only 0.27%.On WideResNet-40 with CIFAR100-C, λ=10 produces a 39.58% error rate, close to BN Stats Adapt’s 41.1%.

E.1. TTA works

This section documents baseline implementations and reproduction settings for comparison methods. The baselines include released TTA implementations and additional parameter-efficient transfer-learning methods.

  • Baseline implementations: The experiments use officially released code for TENT, TTT++, CoTTA, EATA, and NOTE.Additional data-loader or pretrained-model-loader code was added where needed.
  • Baseline implementations: BN Stats Adapt is implemented by placing the model in PyTorch train mode during test-time adaptation.It is treated as a non-training TTA approach.
  • Implementation adjustments: Batch-size differences from the original implementations are handled by linearly scaling the learning rate.The adjustment follows the cited batch-size scaling procedure.
  • Parameter-efficient baselines: TinyTL, Rep-Net, and AuxAdapt are reproduced with a unified ResNet-50 backbone using their papers and published code.TinyTL attaches LiteResidualModules to ResNet-50 layers 1–4.

F. Results of all corruptions

The section reports continual TTA results across all 15 corruptions on CIFAR-C benchmarks, including error rates and total memory consumption.

  • The reported memory measure captures total consumption, including parameter sizes and activation storage.
  • Tables 19 and 20 report continual TTA results for CIFARC10-C and CIFARC100-C at severity level 5.
  • Average error summarizes performance across all 15 corruptions, while memory includes model parameters and activations.
Loading 2303.01904v4…