Source-linked AI summary
On the Limits of Representation-Based Test-Time Adaptation for Multimodal LLMs
Mikołaj Piórczyński, Wojciech Masarczyk, Damian Sójka, Sebastian Cygert, Bartłomiej Twardowski
TL;DR
Common corruptions degrade vision-language models, while test-time adaptation for generative MLLMs remains underexplored and must contend with visual-token sequences. The paper evaluates sample-blind representation corrections and finds they recover little corruption-related performance; it links this result to shifts that are nearly orthogonal and token-specific.
Problem
Whether test-time adaptation can recover MLLM performance under common corruptions is underexplored, despite deployment often providing only unlabeled, corrupted inputs.
Source: §1 · pp. 1–2Method
Results
Across three MLLMs and 15 corruptions, representation-based corrections close no meaningful fraction of the accuracy gap, including when fitted offline on paired clean–corrupted activations.
Source: §1 · p. 2Takeaways & Limitations
Nearly orthogonal, token-specific shifts leave little for sample-blind corrections to remove, so effective restoration would need to vary by sample and token.
Source: §1 · p. 2The study covers three 2–4B models from one language-model family, one 765-question multiple-choice benchmark, encoder/projector corrections, and synthetic single-severity corruptions.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Vision-language models suffer significant accuracy drops under common corruptions. While test-time adaptation (TTA) successfully mitigates these shifts in CLIP-like contrastive models, its application to generative multimodal large language models (MLLMs) remains underexplored and uniquely challenging due to their sequential token generation. In this work, we demonstrate that representation-based TTA fails to recover performance in MLLMs. Our experiments reveal that even when violating standard TTA assumptions by leveraging target data offline, sample-blind feature alignment recovers only marginal accuracy. We attribute this failure to the nature of corruption shifts: unlike CLIP-like architectures, which benefit from a shared domain-shift direction within a single pooled classification embedding, MLLM visual token sequence shifts are nearly orthogonal and span the full embedding space.These findings bound what any sample-blind, position-shared correction at a single encoder site can achieve, establishing that representation-space interventions alone might be insufficient to ensure corruption robustness in MLLMs.
1 Introduction
Common corruptions degrade vision-language models, but whether test-time adaptation can recover MLLM performance remains underexplored. The paper finds sample-blind representation corrections provide little recovery, attributing this to token-specific shifts that differ from the pooled representations used by classifiers.
1 Introduction
- Common corruptions such as sensor noise, defocus, and brightness shifts degrade VLM accuracy even when scenes remain recognizable.At deployment, only unlabeled, already-corrupted inputs are available.Source: §1 · pp. 1–2
- Representation-based TTA applies one sample-blind correction estimated from activation statistics, but MLLMs require restoring a visual-token sequence rather than one pooled image representation.These methods need neither gradients nor labels and do not assume a fixed output space.Source: §1 · p. 2
- Across three MLLMs and 15 corruptions, representation-based corrections close no meaningful fraction of the corruption gap, even when fitted offline on paired clean and corrupted activations.The paper contrasts this with a linear probe, for which matching pooled projector activations recovers most of the gap.Source: §1 · p. 2
- Corruption-induced shifts are nearly orthogonal across images and span almost the full embedding space, leaving little for a shared sample-blind correction to remove.Effective restoration would need to vary by sample and by token.Source: §1 · p. 2
- The paper presents a first evaluation of MLLM test-time adaptation for common corruptions across three models and 15 corruptions, alongside a quantitative bound on removable shift.The reported representation-based corrections yield no practically significant gains.Source: §1 · p. 2
2 Related Work
Prior work documents MLLM fragility under corruptions and develops TTA primarily for classifiers. Existing MLLM adaptation work does not evaluate corruption robustness using only unlabeled corrupted test inputs, motivating the paper’s study of activation-based correction.
2 Related Work
- Existing corruption benchmarks for MLLMs diagnose substantial degradation but do not propose remedies, while prior mitigation approaches have deployment constraints.MAA requires paired clean–corrupted training data and multiple epochs of adapter training.Source: §2 · p. 2
- To the authors’ knowledge, no prior work adapts general-purpose MLLMs to corrupted images directly at test time using only unlabeled test samples.The paper positions this as the gap it addresses.Source: §2 · p. 2
- Most TTA methods target classifiers, using feature alignment or backpropagation-based objectives that transfer poorly to MLLMs’ open-ended outputs and deployment scale.Entropy and pseudo-label methods presuppose a fixed label space and calibrated uncertainty; online updates also risk collapse and forgetting.Source: §2 · p. 3
- Prior work on MLLM test-time adaptation aggregates predictions from augmented inputs and may update parameters, but does not evaluate common-corruption robustness.The cited approach targets general test-time scaling for small MLLMs.Source: §2 · p. 3
- Activation steering modifies activations rather than weights, and prior MLLM uses target high-level semantic behavior; AcT frames steering as transport between activation distributions.The paper considers Gaussian-AcT, which recovers NEO and FOA re-centering and PEA’s whitening–coloring transform as special cases.Source: §2 · p. 3
3 Experiments
The experiments test sample-blind corrections to visual-encoder activations in three MLLMs on RealWorldQA under 15 severity-5 corruptions. Online gains are marginal, and offline oracle tests show that greater map expressivity or supervision does not remove the core limitation.
3 Experiments
- The experiments edit frozen vision-encoder activations in Qwen2.5-VL-3B-Instruct, InternVL3-2B, and Ovis2-4B on 765 RealWorldQA questions under 15 severity-5 corruptions.Each correction is estimated from images alone and applies to every question about an image.Source: §3 · p. 3
3.1 Online adaptation
- +1.1 pp on Qwen2.5-VL-3B, +1.0 pp on InternVL3-2B, and +1.6 pp on Ovis2-4B with dynamic weighting recover only 12%, 9%, and 15% of corruption losses.Weighting concentrates on the earliest layers and improves accuracy only gradually as the stream progresses.Source: §3.1 · p. 4
3.2 The method is not the bottleneck
- The oracle MLP recovers at most a quarter of the held-out gap, and WCT stays within one standard error of Gaussian-AcT despite its full covariance.The oracle MLP reaches its best recovery only at a shallow layer; the ridge map lowers accuracy at most layers despite paired supervision.Source: §3.2 · pp. 5–6, Figure 2 · p. 5
- A token’s shift is not recoverable from that token alone; undoing it requires information from the rest of the image.This places the problem closer to restoration than to feature alignment.Source: §3.2 · p. 6
- A perfect per-token correction at one intervention layer would close the gap, but patching individual encoder sub-modules usually recovers no more than about half of it.The exception is Ovis2’s projector-side MLP; the best sub-module location depends on the architecture.Source: §3.2 · p. 6
4 Analysis
The analysis finds that corruption shifts are largely token-specific, leaving little shared structure for sample-blind corrections to remove. The degradation is also distributed broadly across patches and feature dimensions, so recovery requires restoring a substantial part of the representation.
4.1 The shift is token-specific
- Within-image token residuals dominate the shift, which becomes nearly orthogonal across images with depth and spans almost the full embedding space, unlike pooled shifts.The decomposition separates global, position-specific, sample-specific, and within-image token variation; pooled averaging suppresses the residual that remains visible to the language model.Source: §4.1 · pp. 6–7
- For pooled clean-feature probes, corruption lowers accuracy from 94.5% to 57.6%, and Gaussian-AcT raises it to 84.0%; tokenwise mapping recovers ρ = −0.04 to 0.04 on RealWorldQA.The pooled probe improvement occurs because matching the first two moments makes scene information linearly decodable again; the tokenwise map does not recover the MLLM accuracy gap.Source: §4.1 · p. 7
4.2 The degradation is not localized
- An oracle intervention replaces random subsets of final-block corrupted activations with clean activations from the same image, sweeping patch positions and feature dimensions.The experiment tests Gaussian noise at severity 5 across all three models.Source: §4.2 · p. 8
- Half of the clean–corrupted gap requires restoring roughly one-third of patches or 60% of feature dimensions, and dimension recovery stays below the proportional rate.Patch recovery is concave and begins immediately, while dimension recovery is flat below about 20% of dimensions.Source: §4.2 · p. 8, Figure 5 · p. 8
5 Discussion, Limitations, and Future Work
Across three MLLMs and 15 corruptions, representation-based TTA recovers little of the accuracy loss, which the paper relates to shifts that are mostly token-specific and not concentrated in small representation subsets. The discussion limits the study to sample-blind corrections and leaves open whether token-level corrections can be learned from unlabeled test streams.
5 Discussion, Limitations, and Future Work
- Across three MLLMs and 15 corruptions, representation-based TTA recovers about one percentage point of a much larger drop, while tested variants recover at most a few points.The authors characterize adaptation as a per-token restoration problem rather than feature alignment, and do not expect sample-blind corrections to repair corruption damage at deployment.Source: §5 · p. 8
- The study covers three 2–4B models from one language-model family, one 765-question multiple-choice benchmark, encoder/projector interventions, and synthetic single-severity, single-corruption streams.The authors note that these streams differ from mixed degradations encountered in deployment.Source: §5 · p. 8
- Whether a test-time method can learn token-level corrections from an unlabeled stream without paired clean–corrupted data remains open.The paper points to consistency across augmentations as one possible approach.Source: §5 · p. 8
A Experimental Details
Experiments intervene at encoder-block outputs and model-specific visual projection sites selected to span encoder depth. The adaptation uses precomputed clean-source activation statistics, while target statistics are estimated offline or online.
A Experimental Details
- Interventions span encoder depth at model-specific block outputs and the projector or visual embedding table; single-layer experiments use activation-patching-selected blocks.The listed blocks are approximately evenly spaced by relative depth.Source: A Experimental Details · p. 12
- Accuracy is averaged over three random seeds, which also determine online-adaptation sample order.Target statistics are estimated either offline on the full test set or online from the test stream.Source: A Experimental Details · p. 12
- Clean-source activation statistics are precomputed from a calibration set and retained instead of source examples, with a small fixed storage cost.This relaxes settings that prohibit retaining any source-derived information; the adaptation procedure never sees a source example.Source: A Experimental Details · p. 12
B Online Estimation and Dynamic Weighting
Target moments are estimated from the incoming stream, while dynamic layer weights account for both distribution discrepancy and how much data supports each estimate. The reported stream curves show that adaptation remains small and that the weights concentrate on early layers.
B Online Estimation and Dynamic Weighting
- Target moments update after batches of 16 samples using an exponential moving average with rate 0.1.The updates track running means and variances for every intervention layer.Source: B Online Estimation and Dynamic… · p. 12
- Dynamic weighting rescales each layer’s Gaussian-fit KL discrepancy to [0, 1], then discounts layers with sparse estimates using nℓ/(nℓ+B).This keeps a layer with little data from receiving full correction strength merely because its estimate appears highly discrepant.Source: B Online Estimation and Dynamic… · p. 12
- The unweighted correction drops sharply early while online estimates remain poor, then recovers to about break-even; the weighted variant stays near zero and ends with a small gain.Figure 6 plots cumulative RealWorldQA accuracy change over the no-TTA baseline along the stream, averaging over corruptions and three orders.Source: Figure 6 · p. 13
- Mean per-layer weight over the stream is plotted against relative encoder depth, with the weight concentrating on early layers.Figure 7 visualizes the dynamically weighted variant’s layer weights.Source: Figure 7 · p. 13
C Ablations of Single-Layer Gaussian-AcT
Single-layer Gaussian-AcT benefits from per-token rather than pooled moments, and online per-token fitting closely matches offline fitting. Activation patching finds that recoverable degradation is distributed differently across architectures, bounding what a single sub-module can repair.
C Ablations of Single-Layer Gaussian-AcT
- Per-token moments yield gains of +0.87 ± 0.38, +1.32 ± 0.23, and +0.91 ± 0.10 pp on InternVL3-2B, Qwen2.5-VL-3B, and Ovis2-4B, respectively, with online fitting close to offline gains.Offline gains are +0.95, +1.27, and +1.05 pp in the same model order; Figure 9 compares per-corruption online and offline gains.Source: C Ablations of Single-Layer Gaussian-AcT · p. 13, Figure 9 · p. 14
- Per-token moments outperform global moments; pooling moments over all token positions reduces the gain to +0.12 ± 0.17, +0.88 ± 0.14, and +0.25 ± 0.08 pp.Global estimates converge within 50 samples but plateau, whereas per-token estimates converge after 200–300 samples and reach offline-fitting accuracy.Source: C Ablations of Single-Layer Gaussian-AcT · p. 13, Figure 8 · p. 14
D Activation Patching: Where the Degradation Can Be Repaired
- Activation patching replaces one encoder attention or MLP branch with its clean-image activation; because this is the best possible correction there, its recovered margin upper-bounds any adaptation confined to that sub-module.Figure 10 maps the share of the clean–corrupted margin gap recovered across 15 severity-5 corruptions and encoder blocks.Source: D Activation Patching: Where the… · p. 14
- Repair depends on architecture: Qwen2.5-VL-3B has early-MLP and attention hubs, Ovis2-4B shows broader repair, and InternVL3-2B spreads recoverable damage thinly across mid-depth blocks.Except for Ovis2-4B’s projector-side MLP, no single sub-module recovers more than about half the gap; that MLP’s effect may reflect restoration of the visual stream as a whole.Source: D Activation Patching: Where the… · p. 15
- The single-layer intervention is placed at the upstream edge of a mid-depth region with corruption-general attention repair, trading off against stronger but corruption-specific or late-encoder components.This placement offers more single-block headroom in Qwen2.5-VL-3B and Ovis2-4B than in InternVL3-2B, where repair is spread thinly.Source: D Activation Patching: Where the… · p. 15
E Interpolating Between Corrupted and Clean Activations
The section tests how accuracy changes when visual activations are interpolated from corrupted values toward their paired clean values. For Qwen2.5-VL-3B under Gaussian noise, accuracy rises with interpolation and reaches clean accuracy, though the manifold conclusion is indirect and narrowly tested.
E Interpolating Between Corrupted and Clean Activations
- The interpolation sweeps α from 0 (corrupted activation) to 1 (clean activation) at visual-token outputs of encoder blocks 4, 18, and 30 and the projector.The sweep measures downstream accuracy independently of an estimator’s correction error.Source: E Interpolating Between Corrupted and… · p. 15
- For Qwen2.5-VL-3B, accuracy grows roughly in proportion to interpolation weight at every tested site, with no intermediate value below the no-TTA baseline and clean accuracy reached at α = 1.The sweep uses 200 paired clean and corrupted samples under severity-5 Gaussian noise; the authors infer the straight path remains within a region the model reads correctly.Source: E Interpolating Between Corrupted and… · p. 15
- Figure 10’s repair map encodes corruptions as severity-5 rows colored by family and encoder blocks as relative-depth columns, separately for attention and MLP patches.Each cell averages 100 images, and the color scale saturates at ±0.5.Source: Figure 10 · p. 16
- The manifold interpretation is limited: membership is inferred from accuracy along the path, and the sweep covers only one model and one corruption.Source: E Interpolating Between Corrupted and… · p. 16
F Linear Probe on Pooled Projector Activations
A linear probe on pooled projector activations shows that Gaussian-AcT can restore linearly decodable scene information, but the same position-shared transport does not recover multiple-choice answers. The section also reports limits on the probe labels and on which units can be selected for restoration.
F Linear Probe on Pooled Projector Activations
- The probe experiment uses five scene pseudo-classes formed by clustering clean pooled projector activations, with the three model partitions merged by co-association consensus.The models agree on the partition with ARI 0.73; held-out probe evaluation uses a stratified 70/30 split.Source: F Linear Probe on Pooled Projector… · p. 16
- Severity-5 corruption lowers pooled-projector probe accuracy from 94.5% to 57.6 ± 0.3%, while Gaussian-AcT lifts it to 84.0 ± 0.2% and recovers ρ = 0.65–0.78 of the gap.Across 45 model–corruption pairs, the probe improves all pairs and ρ exceeds 0.6 for 13 of 15 corruptions, peaking at 0.95 for InternVL3-2B under defocus blur.Source: F Linear Probe on Pooled Projector… · p. 17, Table 2 · p. 18
- For multiple-choice answers, the same offline projector map recovers ρ = −0.04, 0.04, and −0.01 of the RealWorldQA accuracy gap for InternVL3-2B, Qwen2.5-VL-3B, and Ovis2-4B.Unlike the probe, this transport acts on the unpooled token sequence shared across positions and is read by the language model.Source: F Linear Probe on Pooled Projector… · p. 17, Table 2 · p. 18
- Probe labels are derived from clean activations of all 765 images, so absolute accuracies are inflated, while the clean–corrupted drop and recovered fraction ρ retain their direction.The probe and transport use pooled sample vectors, so this experiment diagnoses retained scene information rather than constituting a token-level adaptation method.Source: F Linear Probe on Pooled Projector… · p. 17
G Per-Corruption Decomposition of the Squared Shift
- Figure 12 breaks the squared-shift decomposition down by corruption at a shallow block, identified as the most favorable case for sample-blind adaptation.Source: G Per-Corruption Decomposition of the… · p. 18
H.1 Selecting the substituted subset by importance
- Selecting patches or channels by clean activation magnitude or dataset variation does not change the recovery-curve shape; informed selection rescales it at best, and patch selection is never better than random.Only the highest-ranked channel rules improve on random selection; lowest-ranked channel rules match it.Source: §H.1 · p. 18
H.2 Additional models
Additional-model analyses find that most corruption shifts are not captured by a shared direction or small representation subset, while prompting has no measurable effect on corrupted accuracy.
H.2 Additional models
- Only noise and contrast had a sizable global shift component, up to 0.30; geometric and compression corruptions were almost entirely residual.This decomposition was measured at a shallow block, the most favorable case for sample-blind adaptation.Source: Figure 12 · p. 19
I.1 Prompting does not mitigate image corruptions
- The tested prompt instructed models to ignore visual artifacts and infer underlying content from large-scale evidence rather than fine local texture.The evaluation changed the system prompt while leaving each model’s default message as the baseline.Source: §I.1 · p. 19
- Prompting changed corrupted accuracy by −0.5, +0.5, and −0.4 pp for InternVL3-2B, Qwen2.5-VL-3B, and Ovis2-4B, respectively, with no measurable effect.Clean accuracy was also unchanged, and prompted and unprompted predictions were nearly identical.Source: §I.1 · p. 19
- Importance-based patch or channel selection did not outperform random substitution for Qwen2.5-VL-3B under Gaussian noise at severity 5.Selection ranked units by clean-activation magnitude or across-dataset variance.Source: Figure 13 · p. 20
H.2 Additional models
- For InternVL3-2B and Ovis2-4B, the random-substitution sweeps likewise show that no small representation subset carries the degradation.The figures plot the recovered clean–corrupted gap against the fraction of patches or channels substituted.Source: §H.2 · p. 18, Figure 14 · p. 20, Figure 15 · p. 20
I.2 Confidence is a weak predictor of correctness
The authors test whether answer confidence predicts correctness and find that its signal weakens under Gaussian noise. They explain that multiple-choice answers lack shared class meanings for adaptation to accumulate across questions.
I.2 Confidence is a weak predictor of correctness
- AUROC fell from 0.684 (95% CI [0.645, 0.720]) on clean images to 0.588 [0.549, 0.628] under Gaussian noise; 168 flipped answers favored distractors by an average margin of 0.258.The noisy-image AUROC was barely above the chance band, and the ROC figure shows clean and noisy results for Qwen2.5-VL-3B on RealWorldQA.Source: §I.2 · p. 20, Figure 17 · p. 21
- Because option letters change meaning across questions and choice order, multiple-choice VQA offers no persistent class directions for adaptation to accumulate.Prompt tuning can only provide generic instruction or output-format effects, still driven by the weak confidence signal.Source: §I.2 · p. 21