PaperlayerView exact revision on arXiv

Source-linked AI summary

Adapting Without Gradients: Affine Statistics Transport and What Its Certificate Can Tell You

Salim Khazem, Ibrahim Mohamed Serouis

arXiv:2609.00374v1cs.LGcs.AIcs.CV

TL;DR

The paper addresses test-time adaptation when learned models must remain frozen and backward computation or optimizer state may be unavailable. CASTER transports source class statistics through a target-batch affine map, achieving stronger frozen-feature performance than k-NN in most evaluated settings while using a certificate to detect unsupported transport. The certificate improves affine transport on average but is mechanism-specific and does not reliably predict adaptation benefit or transfer to feature-updating methods.

  • Problem

    Test-time adaptation often assumes mutable model parameters, backward computation, or optimizer state, which are unavailable or restrictive in frozen-model deployments.

  • Method

    CASTER stores source class moments in a discriminative subspace, estimates a class-shared affine transformation from target-batch moments, and analytically transports class distributions without updating the network.

  • Results

    27 of 28 backbone-dataset settings favor CASTER over k-NN on identical frozen features, with a median 18× reduction in retained state; gating changes transport from −3.35 points to +1.69 points on average.

  • Takeaways & Limitations

    The certificate is a safety floor for detecting empirically degenerate affine transport, not a general predictor of how much adaptation will help.

  • Takeaways & Limitations

    CASTER assumes a class-shared affine shift, and its certificate is mechanism-specific, with no general guarantee for updates that change the representation.

Abstract

from arXiv · show

Test-time adaptation (TTA) typically assumes that model parameters can be updated at inference time. This assumption is restrictive for inference-only accelerators, frozen or third-party models, and memory-constrained deployments, and standard BatchNorm-based TTA configurations may also become inactive on architectures without BatchNorm. We study adaptation when the learned model must remain frozen. We introduce CASTER, a gradient-free method that stores source class statistics in a discriminative subspace, estimates a class-shared affine transformation from target-batch moments, and analytically transports the source class distributions before classification. CASTER requires no backward pass, optimizer state, or stored source feature bank. Across four backbones and seven datasets, it outperforms k-NN on identical frozen features in 27 of 28 backbone-dataset settings while retaining a median of 18x less state. Affine transport is not always reliable. On ImageNet-C, where batches contain only 64 samples for 1000 classes, unconditional transport loses 21.2 top-1 points. We therefore introduce an empirical residual-to-margin transportability certificate. Across 307 evaluation cells, every transport losing more than 10 points has certificate value above 3.9, although benign and destructive regimes are not perfectly separated. Gating converts an average $-3.35$-point effect of unconditional transport into a +1.69-point gain, and performance remains within 0.3 points of the best threshold over a broad threshold range. Finally, we show that this certificate is mechanism-specific: when applied to Tent, it accepts only $4.3\%$ of updates and preserves 0.6% of Tent's available gain. These results position CASTER as a lightweight adaptation mechanism for frozen-model deployment, together with an explicit account of when its safety signal is informative and when it is not.

1. Introduction

The paper studies test-time adaptation with frozen learned models, targeting deployments where backward computation, mutable parameters, or optimizer state are unavailable. It introduces CASTER for lightweight affine-statistics transport and an empirical certificate for detecting unsupported transport.

  • Motivation and contribution: CASTER performs gradient-free test-time adaptation while keeping the feature extractor and learned parameters unchanged.It stores source class moments, estimates a class-shared affine map from target-batch moments, and classifies transported Gaussian distributions.
  • Failure regime: 21.2 top-1 points are lost by unconditional transport on ImageNet-C with 64 samples across 1000 classes.Sparse pseudo-class coverage makes batch-level class geometry unreliable in this regime.
  • Certificate: Every admissible evaluation cell losing more than 10 points has certificate value above 3.9.Across a roughly fourfold threshold interval, performance remains within 0.3 points of the best threshold, but the certificate does not rank adaptation benefit.
  • Certificate: The certificate admits only 4.3% of Tent updates and preserves 0.6% of Tent’s available gain.Its failure on Tent is structural because Tent changes the representation that the certificate evaluates before updating.
  • Motivation and contribution: 27 of 28 backbone-dataset cells favor CASTER over k-NN on identical frozen features.The method also retains an 18× smaller median state and remains applicable when evaluated BatchNorm-only Tent/EATA configurations are inert.

2. Related Work

Related work spans gradient-based, backward-free, frozen-model, selective-monitoring, and robustness approaches to test-time adaptation. CASTER differs by targeting geometrically unsupported affine transport rather than generic predictive risk or distribution distance.

  • Gradient-based TTA: Tent, EATA, CoTTA, and SAR adapt models through gradient-based parameter or normalization updates.These methods address online adaptation when mutable model state and backward computation are available.
  • Backward-free and frozen-model adaptation: T3A and FOA provide backward-free alternatives while retaining fixed or largely frozen learned representations.T3A uses confident target pseudo-prototypes; FOA optimizes an added input prompt with a derivative-free strategy.
  • Backward-free and frozen-model adaptation: k-NN over identical frozen features is a strong nonparametric baseline whose retained state grows with the number of source examples.This contrasts with CASTER’s stored class-statistics representation.
  • Selective adaptation and risk monitoring: Selective prediction and TTA risk monitoring ask whether a model should act under a confidence or predicted-risk criterion.The paper’s narrower question is whether analytic transport is geometrically unsupported before use.
  • Benchmarks and distances: CIFAR-C, ImageNet-C, ImageNet-V2, ImageNet-R, ImageNet-Sketch, PACS, and Office-Home benchmark corruption or domain shifts.MMD and Wasserstein distance quantify distribution separation, which is distinct from expected benefit of adaptation.

3. Method

CASTER builds a discriminative subspace, stores class-level source moments, estimates a target-batch affine map, and transports source class distributions for Gaussian discriminant prediction. It uses sample-dependent covariance simplification and rejects batches whose geometry lacks sufficient empirical support.

  • Discriminative subspace: CASTER constructs a discriminative subspace from generalized eigenvectors of between-class and within-class scatter matrices.The active dimensionality is bounded by both class structure and target-batch support.
  • Discriminative subspace: Source statistics are stored in the largest subspace and restricted to the first k coordinates for each target batch.The B−1 bound avoids unsupported target-covariance directions, so retained state does not depend on source-example count.
  • Source statistics: For each class, CASTER stores a projected source mean, covariance, and prior.Global source mean and covariance are also maintained in the active subspace.
  • Affine statistics transport: CASTER estimates a shrinkage-stabilized whiten-color map from target-batch empirical mean and covariance.When B < 2k, it switches from a dense O(k^2) map to a diagonal O(k) analogue as a sample-efficiency safeguard.
  • Affine statistics transport: The affine map transports source class statistics, which are then used by a Gaussian discriminant without changing network parameters.Class-covariance regularization is applied during transport.
  • Transportability certificate: CASTER checks transportability before committing to target predictions.It retains confident samples, inverse-transports them, and computes centroid residuals relative to source-class separation.
  • Transportability certificate: The transported classifier is accepted only when the residual threshold and minimum coverage requirements are satisfied.Otherwise, the batch is rejected and the frozen model head is used; thresholds are fixed globally.
  • Transportability certificate: Using the same Mahalanobis metric for residual and margin makes the certificate dimensionless and invariant to invertible feature rescaling.This metric consistency is presented as necessary for transport-quality measurement independent of feature conditioning.

4. Experiments

The experiments evaluate frozen-feature estimators and corruption adaptation under controlled protocols, with explicit safeguards for checkpoint validity, reproducibility, and measurement against an immutable source model.

  • Frozen-feature estimation: Four ImageNet-pretrained backbones and seven datasets are used to evaluate estimators on identical frozen features.The backbones remain frozen without task-specific finetuning, so estimator differences are measured on the same representations.
  • Corruption benchmarks: CIFAR-10-C and CIFAR-100-C are evaluated across 19 corruptions and 5 severities over six backbones.Runs require an existing checkpoint, clean-source accuracy above the stated sanity threshold, and source moments from that exact checkpoint.
  • Reproducibility: The verification script fails closed when required artifacts are missing or inconsistent.ResNet-50 uses three seeds, while deterministic or single-seed backbones are reported without inventing seed replication.
  • Measurement conventions: Harm is measured against an immutable unadapted source model, and inactive baselines must reproduce source-only inference exactly.This avoids making harmful updates appear benign or manufacturing differences through train-mode behavior.

4.2. Frozen-Feature Estimation

The study first compares CASTER with k-NN on identical frozen features, then examines when transport is safe and what its certificate can support.

  • Frozen-feature estimation: 27 of 28 backbone-dataset cells favor CASTER over k-NN on identical frozen features.The only exception is ViT-B/16 on CIFAR-10, where the methods differ by less than 0.04 top-1 points; Flowers-102 shows gains of approximately 18–22 points on three backbones.
  • State cost: 781 MiB is reached by the k-NN bank on Tiny-ImageNet, whereas CASTER stores class moments without dependence on source-example count.The storage advantage is not universal: on Flowers-102, k-NN stores only 1020 training images and CASTER’s covariance state is comparatively large.
  • Certificate scope: The certificate supports catastrophe detection but not benefit ranking.Every admissible cell losing more than 10 top-1 points has certificate value above 3.9, while safe and unsafe bands overlap.
  • Certificate scope: Within the non-degenerate regime, the certificate score is positively correlated with transport gain, so it should detect unsupported transport rather than rank utility.On CIFAR-10-C, where catastrophic transport is absent, thresholding does not beat transporting every batch.
  • Thresholding: −3.35 points from unconditional transport becomes +1.69 points with gating, while thresholds from 1.0 to 3.9 remain within 0.3 points of the best observed value.The shipped operating point is τ = 0.8.

4.4. What the Gate Buys

The gate is intended to preserve useful transport while avoiding destructive updates, and its aggregate benefit reflects that asymmetric trade-off.

  • Destructive regimes: +10.2 points on CIFAR-100-C and +21.2 on ImageNet-C are recovered relative to always transporting.On ImageNet-C every batch is rejected, so CASTER matches the frozen head.
  • Helpful regimes: 0.1 points on Flowers-102, 0.2 on Oxford-IIIT Pets, and at most 2.2 on CIFAR-10-C are sacrificed where transport helps.No fixed policy dominates across all regimes.
  • Certificate visualization: Figure 3 places certificate score against realized transport gain for one point per admissible evaluation cell on i.i.d. streams.The symmetric-log axis exposes both benign and catastrophic regimes, while losses above 10 points occur above 3.9.
  • Normalization grouping: Table 1 aggregates corruption results over 19 corruptions and 5 severities, grouping results by BatchNorm and LayerNorm backbones.Tent and EATA reproduce the frozen head exactly on the four LayerNorm backbones under their standard BatchNorm-only parameter collection.
  • Aggregate policy comparison: 60.05 top-1 is achieved by certificate gating, compared with 55.70 for always transporting and 59.05 for never transporting.These scores are pooled over admissible ResNet-50 cells.

4.5. Corruption Benchmarks and Normalization

Corruption experiments distinguish normalization-dependent baselines from feature-space adaptation and show that low class coverage can make affine transport unsafe.

  • Normalization: Tent and EATA are inert on the four LayerNorm backbones because their evaluated BatchNorm-only parameter sets are empty.SAR remains active because it can adapt LayerNorm parameters, while CASTER operates on frozen extracted features independently of normalization type.
  • Normalization: The benchmark does not claim entropy minimization is impossible on LayerNorm networks; it identifies inactivity in the canonical BatchNorm-parameterized configurations evaluated.Separating normalization families prevents inactive cells from being mistaken for evidence about active adaptation.
  • Low-coverage stress test: ImageNet-C has 1000 classes but only 64 samples per target batch, leaving batch-level class geometry unreliable when most classes are absent.This is a low-coverage stress test for affine class-statistics transport, not evidence that the frozen representation is unusable in principle.
  • ImageNet-C results: Relative to the frozen head, Source-GDA loses 19.0 points, the non-transported Gaussian estimator loses 19.0 points, unconditional transport loses 21.2, and T3A loses 33.5.SAR improves over the frozen head by +4.6 points in this experiment.
  • Gated transport: The certificate rejects every ImageNet-C target batch, so deployed CASTER matches the frozen head and incurs zero additional harm.The policy abstains when observed batches do not support the assumptions required by affine class-statistics transport.

4.7. Certificates Do Not Transfer Across Mechanisms

The transportability certificate is effective for CASTER’s affine transport but does not transfer safely to Tent or other mechanisms that change the representation.

  • The certificate leaves Tent essentially indistinguishable from the frozen model, while a more permissive referee still discards roughly two thirds of Tent’s available improvement.
  • On 57 matched CIFAR-10-C cells, the certificate accepts only 4.3% of Tent updates and preserves 0.6% of Tent’s available gain.EATA exhibits the same qualitative behavior as Tent.
  • Neither criterion is a useful safety layer for Tent.
  • Certificate acceptance is strongly anti-correlated with Tent’s ungated gain (ρ = −0.86), unlike the positive relationship observed for CASTER’s own transport.
  • The failure is structural because the certificate evaluates pre-update geometry, whereas Tent changes normalization parameters and therefore changes the evaluated representation.

4.8. Cost

Controlled measurements show that CASTER reduces adaptation cost while retaining most frozen-model throughput.

  • CASTER processes 1076 images/s, compared with 667 for Tent, 649 for EATA, and 386 for SAR.These measurements use CIFAR-10-C fog at severity 3 with ResNet-50 on an otherwise idle device.
  • CASTER uses 1249 MiB peak memory, or 22.7% of the 5510 MiB used by gradient-based methods, while retaining 90% of frozen-model throughput.
  • Relative timings from the large sweep are unreliable because jobs shared GPUs; the reported measurements therefore use isolated runs.SAR remains most expensive because its sharpness-aware update requires two forward–backward evaluations per adaptation step.

5. Discussion and Limitations

CASTER targets frozen-model deployments but inherits the frozen representation and assumes a class-shared affine shift, with no safety claim for arbitrary target distributions.

  • Gradient-based TTA remains competitive when model updates are available, and EATA is marginally higher than CASTER on the CIFAR-10-C BatchNorm aggregate.CASTER is higher on the CIFAR-100-C BatchNorm aggregate, so the paper does not claim universal advantage.
  • CASTER cannot repair a frozen representation that does not already separate target classes sufficiently well.Transporting source statistics cannot recover information absent from the representation.
  • Comparisons with k-NN use identical extracted features because the representation limitation is shared by frozen-feature estimators.
  • The class-shared affine assumption can fail under strongly class-dependent shifts or when target batches are too small or imbalanced for reliable empirical moments.Sparse class composition can be confounded with distribution shift.
  • The certificate is a mechanism-specific diagnostic rather than a formal guarantee, and certificates for other mechanisms must be derived and validated separately.

6. Conclusion

CASTER provides gradient-free adaptation for frozen-model constraints and combines strong frozen-feature performance with an empirical gate for unreliable affine transport.

  • On identical frozen features, CASTER outperforms k-nearest neighbours in 27 of 28 backbone–dataset settings while reducing retained state by a median 18×.
  • CASTER estimates a class-shared affine map from target-batch moments and requires neither a backward pass nor a stored source feature bank.
  • Unconditional transport changes accuracy by −3.35 points relative to the frozen head, whereas gating at the shipped threshold yields +1.69 points.
  • A roughly fourfold threshold range remains within 0.3 points of the best observed operating point.
  • Applied to Tent, the certificate accepts only 4.3% of updates and preserves 0.6% of Tent’s available gain, demonstrating mechanism specificity.
Loading 2609.00374v1…