PaperlayerView exact revision on arXiv

Source-linked AI summary

Test-Time Prompt-Agnostic Decomposition

Junze Wang, Lei Fan, Dezheng Zhang, Donglin Di, Yang Song, Sidong Liu, Cong Cong

arXiv:2610.35090v1cs.CV

TL;DR

TPT adapts pretrained models to shifted test distributions by updating prompts, but locally optimized updates can accumulate across steps and destabilize prompt trajectories. TPD decomposes update magnitude and direction and applies complementary controls; across 15 datasets, it improves 10 TPT methods, with gains for both visual and text prompts.

  • Problem

    Existing TPT methods mainly optimize each update locally, while reused prompts let noisy or biased unlabeled objectives accumulate across test steps.

    Source: §1 · p. 1

  • Method

    TPD analyzes base prompt-update trajectories by magnitude and direction, using AKC to shrink updates under expansion and HUR to preserve persistent while suppressing oscillatory and residual directions.

    Source: §1 · pp. 2–3

  • Results

    Across 10 TPT methods on 15 datasets, TPD gains 3–8% accuracy for visual prompts and over 2% for text prompts.

    Source: §1 · p. 3

  • Takeaways & Limitations

    TPD serves as a plug-in stabilizer for visual and text TPT without modifying the backbone, prompt architecture, or adaptation loss.

    Source: §1 · p. 2, §5 · p. 10

    TPD may be less effective with sparse, noisy, or rapidly changing update histories, and its finite-window local model does not guarantee global convergence or semantic correctness.

    Source: §5 · p. 10

Abstract

from arXiv · show

Test-Time Prompt Tuning (TPT) adapts large pretrained models to unseen target distribution shifts by updating only lightweight prompt tokens. Existing TPT methods are effective, but they mainly optimize each prompt update locally. Across test-time steps, prompts updated at the current step are reused for the corresponding predictions and updates, so noisy or biased unlabeled objectives can accumulate and destabilize the prompt-update trajectory. We identify two failure modes: magnitude expansion, where updates move the prompt too far, and directional drift, where consecutive updates point in inconsistent directions. To analyze and control these failures, we propose Test-Time Prompt-Agnostic Decomposition ($\mathtt{TPD}$), which characterizes observed prompt-update trajectories from a dynamical-systems view and decomposes base prompt updates into magnitude and direction. $\mathtt{TPD}$ computes a spectral radius to measure update expansion and decomposes update directions into persistent, oscillatory, and residual components. Building on this decomposition, Adaptive Koopman Control ($\mathtt{AKC}$) regulates update magnitude by shrinking updates under expansive recent dynamics, while Hankel Update Router ($\mathtt{HUR}$) refines update direction by preserving persistent components and suppressing oscillatory and residual components. Together, $\mathtt{AKC}$ and $\mathtt{HUR}$ produce a stabilized prompt update. As a prompt-agnostic framework, $\mathtt{TPD}$ can be plugged into visual and text TPT methods without modifying the backbone, prompt architecture, or adaptation loss. Experiments on 15 datasets show that $\mathtt{TPD}$ consistently improves 10 baselines, with accuracy gains of $3$--$8\%$ for visual prompts and over $2\%$ for text prompts.

1 Introduction

Existing TPT methods optimize prompt updates locally, although updates are reused across test steps and noisy or biased objectives can accumulate into unstable trajectories. TPD analyzes these trajectories by separating update magnitude and direction, then stabilizes them with complementary controls.

1 Introduction

  • Because each step's prompt is reused for predictions and updates, noisy or biased unlabeled objectives can accumulate across test steps.This cross-step dependency can yield magnitude expansion, inconsistent update directions, or both.Source: §1 · pp. 1–2
  • TPD characterizes prompt-update trajectories through spectral-radius expansion measures and persistent, oscillatory, and residual direction components.The persistent component captures consistent adaptation signals, the oscillatory component captures sign-flipping interference, and the residual captures unexplained noise.Source: §1 · p. 2
  • AKC shrinks updates under expansive recent dynamics, while HUR preserves persistent directions and suppresses oscillatory and residual components.Together, the mechanisms produce a stabilized prompt update.Source: §1 · pp. 2–3
  • TPD operates on base prompt updates and plugs into visual and text TPT methods without modifying the backbone, prompt architecture, or adaptation loss.The paper presents it as a prompt-agnostic framework for analyzing and stabilizing trajectories across ViT-based visual and CLIP-based text TPT.Source: §1 · p. 2
  • Across 15 datasets and 10 TPT methods, TPD improves accuracy by 3–8% for visual prompts and over 2% for text prompts.The experiments span shifted-domain and cross-dataset benchmarks.Source: §1 · p. 3

2 Related Work

Prior work covers episodic and online test-time prompt adaptation, as well as prompt optimization during training. TPD instead focuses on diagnosing and controlling prompt-update trajectories across test steps within a target-domain adaptation process.

2 Related Work

  • Episodic TPT adapts prompts independently for each sample or episode, while online TPT updates prompts continuously across test steps [Abdul Samadh et al., 2023, Yoon et al., 2024] [Ma et al., 2023, Zhang et al., 2024, Xiao et al., 2025].Episodic approaches include calibration, sample filtering or augmentation selection, and black-box update rules; online approaches regulate drift or use prompt history.Source: §2 · p. 3
  • CTPT targets continual multi-domain prompt retention, whereas DePT addresses domain adaptation [Gao et al., 2022]; TPD controls trajectories within single-target-domain adaptation.The distinction is the adaptation setting and scope, not merely whether prompts are updated over time.Source: §2 · p. 3
  • Prompt-optimization work improves prompt construction, backbone interaction, or training-time optimization, while TPD addresses test-time prompt-update trajectories [Xiao et al., 2026a] [Xiao et al., 2026b,c, Lan and Tian, 2024, Lan et al., 2026].PAE uses dynamical systems for training-time visual prompt optimization, modeling cross-layer evolution [Wang et al., 2026].Source: §2 · p. 3
  • The related-work distinction is trajectory-level diagnosis and control across test steps, rather than improvements to training-time prompt optimization.The paper frames test-time prompt adaptation as comparatively unexplored in this line of work.Source: §2 · p. 3

3 Method

TPD analyzes recent prompt-update trajectories in a short-window Hankel subspace, separating magnitude expansion from directional components. AKC controls update magnitude and HUR routes direction before the combined update is applied to prompts.

3.1 Test-Time Prompt-Agnostic Decomposition

  • TPD delay-embeds recent base updates into a short-window Hankel subspace, using spectral radius for expansion and persistent, oscillatory, and residual components for direction.The Hankel window uses the latest W updates, each embedding K consecutive updates, with W = K + M − 1.Source: §3.1 · pp. 4–5

3.2 Adaptive Koopman Control

  • AKC preserves updates below the stability threshold and scales them down otherwise, with a gain floor that keeps updates small across consecutive unstable steps.The gain is bounded below by gmin, so the controlled spectral radius is not always below the threshold.Source: §3.2 · p. 5

3.3 Hankel Update Router

  • HUR preserves the persistent component while attenuating oscillatory and residual components, then extracts the routed update’s latest prompt-update block.The attenuation weights remain nonzero because oscillatory and residual components may still contain useful corrections.Source: §3.1 · p. 5, §3.3 · p. 6

3.4 Controlled Prompt Update

  • Combining AKC’s magnitude control with HUR’s direction routing yields the prompt update applied at the current step, while preserving the original TPT update rule.Within the Hankel decomposition, routing does not reverse or amplify projected coordinates, and transmits the persistent component with coefficient 1.Source: §3.4 · p. 6

4 Experiments

Across visual and text prompt-tuning methods, TPD improves benchmark accuracy and is examined through stabilization comparisons, design ablations, trajectory analyses, and robustness tests. The experiments cover domain generalization and cross-dataset transfer, with additional analyses of calibration, batch-size sensitivity, and update stability.

4.1 Experimental Setup

  • TPD is evaluated on image classification under domain generalization and cross-dataset transfer, using ViT visual and CLIP text prompt-tuning baselines.The setup uses ImageNet-pretrained backbones and reports original baseline results alongside gains from adding TPD.Source: §4.1 · pp. 6–7

4.2 Results

  • Visual-prompt OOD mean gains are 5.51–8.18% and cross-dataset mean gains are 3.44–8.23%; text-prompt gains are 2.27–2.69% and 2.52–2.98%, respectively.The comparisons cover four ImageNet variants for domain generalization and 10 downstream datasets for cross-dataset transfer.Source: Table 1 · p. 7, Table 2 · p. 7, §4.2 · p. 8

4.3 Ablation Study

  • TPD exceeds simple stabilization heuristics: VPT DG accuracy rises from 39.66 with low-pass filtering to 47.84 with TPD, while TPT CSD accuracy rises from 66.23 to 68.03.Per-step overhead is 1.13× on VPT and 1.08× on TPT.Source: §4.2 · p. 8, Table 3 · p. 8
  • The diagonal Koopman mode outperforms the full mode, improving VPT-DG from 43.58 to 45.92 and TPT-CSD from 66.38 to 67.21.The study attributes this advantage to more reliable scalar coefficients under a short Hankel window.Source: §4.3 · p. 8, Table 4 · p. 8
  • Hankel projection improves over plain projection, and removing any HUR routing component underperforms full routing across VPT and TPT.The results support delay embedding and routing that preserves persistent directions, attenuates oscillatory directions, and suppresses residual perturbations.Source: §4.3 · pp. 8–9, Table 4 · p. 8

4.4 Analysis

  • Visual trajectories under TPD show increased persistent ratios, reduced oscillatory and residual ratios, and spectral radii closer to the stable regime; text prompts evolve more smoothly, with AKC keeping radius nearer its stable reference.For text prompts, accuracy gain generally rises from low to moderate persistent ratios before peaking.Source: §4.4 · p. 9, Figure 3 · p. 9, Figure 4 · p. 9

4.5 Robustness and Reliability

  • TPD lowers ECE in seven of eight prompt-method and dataset comparisons; for TPT, ImageNet-A ECE falls from 16.42% to 12.27%.C-TPT on ImageNet-R is the sole exception among the eight comparisons.Source: §4.5 · p. 10, Figure 5 · p. 10
  • On ImageNet-A, TPD improves VPT accuracy in six of seven batch-size settings, while the stable-update fraction increases from 80.07% at B = 8 to 91.16% at B = 128.Accuracy decreases at B = 8, and absolute accuracy remains low at very small batch sizes.Source: §4.5 · p. 10, Figure 5 · p. 10

5 Conclusion

TPD decomposes prompt updates into magnitude and direction to characterize and stabilize trajectories, and improves visual and text TPT baselines across domain-generalization and cross-dataset settings. Its spectral account separates magnitude-based expansion from phase-based direction routing.

5 Conclusion

  • TPD decomposes base prompt updates into magnitude and direction, then uses AKC and HUR to constrain magnitudes and route persistent, oscillatory, and residual components.It operates directly on base updates without modifying the backbone, prompt architecture, or adaptation loss.Source: §5 · p. 10

A Spectral Justification of the Three-Component Decomposition

  • The local Koopman spectrum separates the roles of magnitude and phase: eigenvalue magnitude diagnoses expansion, while phase distinguishes persistent from oscillatory dynamics.AKC uses the spectral radius for magnitude control; HUR uses the direction partition.Source: A Spectral Justification of the… · p. 15
  • In the diagonal model, positive coefficients indicate same-sign continuation and negative coefficients indicate sign-flipping oscillation; unexplained content is residual.Soft gates route observed updates into the three components, keeping routed content proportional to the current update rather than synthesizing it from a prediction.Source: A Spectral Justification of the… · pp. 15–16

B Proof of Lemma 1

The proof supports controlling the local spectral radius by scaling the estimated dynamics. It also motivates a coordinate-wise diagonal estimator for short-window Hankel models, where a full estimator can be underdetermined and more costly.

B Proof of Lemma 1

  • Scaling the local operator by a nonnegative gain scales its spectral radius, and the chosen gain bounds the resulting radius by max(ρ̄, g_minρ_t).When ρ_t ≤ ρ̄/g_min, the scaled radius is at most ρ̄.Source: B Proof of Lemma 1 · p. 16
  • In the local Hankel regime, the full Koopman estimator can be underdetermined when rank reaches the number of available transition columns, while the diagonal estimator supplies scalar mode coefficients.Those coordinate-wise coefficients support the gates and routing decomposition; the diagonal estimator has per-step cost O(rM), versus O(r^2M + r^3) for the full estimator.Source: B Proof of Lemma 1 · pp. 16–17
  • Under the fixed-design noise model, the full estimator's variance is rσ^2tr(G^-1), while each diagonal coefficient estimate has variance σ^2/G_ii.The full estimator fits coupled cross-coordinate effects; the diagonal estimator estimates only the scalar coefficient used by its gate.Source: B Proof of Lemma 1 · p. 17

C Proof of Theorem 1

The theorem proof establishes bounded routing in Hankel space and corresponding prompt-space bounds. Routing preserves the sign of projected coordinates, does not amplify them, and attenuates oscillatory and residual components relative to persistent ones.

C Proof of Theorem 1

  • The routed Hankel coordinate is w_i,t y_i,t, where 0 ≤ w_i,t ≤ 1, so its sign is preserved and its magnitude is not amplified.The weights are convex combinations of 1, β_c, and β_n.Source: C Proof of Theorem 1 · p. 17
  • The persistent component is transmitted with coefficient 1, while oscillatory and residual components are attenuated by β_c and β_n in prompt space.The proof combines orthogonal Hankel residual structure with the routing weights to establish non-amplification and an induced prompt-update bound.Source: C Proof of Theorem 1 · p. 17

D Full Experiments

The full experiments are framed as a multi-axis stress test of TPD's prompt-agnostic stabilization claim, spanning varied datasets, modalities, models, adaptation settings, protocols, baselines, stabilizers, and ablations.

D Full Experiments

  • The evaluation spans datasets, prompt modalities, backbone families, adaptation modes, distribution-shift protocols, TPT baselines, simple stabilizers, and component ablations.The stated goal is to test whether improvements extend beyond a narrow dataset, prompt type, backbone, or heuristic stabilizer choice.Source: D Full Experiments · p. 17

D.1 Full Experimental Details

The experiments evaluate TPD across domain-generalization and cross-dataset transfer settings, attaching it to visual and text prompt-tuning baselines. They use fixed TPD hyperparameters and retain only update-history statistics across episodic samples.

Datasets

  • TPD is evaluated on 15 datasets spanning domain generalization and cross-dataset transfer.The domain-generalization setting uses ImageNet and four OOD variants; cross-dataset transfer covers ten datasets.Source: §D.1 · p. 18

Baseline Models

  • The evaluation pairs TPD with four ViT-based visual TPT methods and six CLIP-based text TPT methods.For each baseline, the study reports both the original method and its TPD-augmented variant.Source: §D.1 · p. 18

Implementation Details

  • The same TPD hyperparameters are fixed across backbones, with K=3, M=8, AKC threshold ρ̄=1.0, and gain floor gmin=0.1.HUR uses βc=0.25 and βn=0.1 on both ViT and CLIP.Source: §D.1 · p. 18
  • For episodic CLIP adaptation, prompts reset after each episode while TPD retains only lightweight recent-update history, not adapted prompts or optimizer state.The ViT protocol uses one test-time step per sample with SGD and a top-10% confidence selection for entropy minimization.Source: §D.1 · p. 18

D.2 Full Per-Dataset Results

The full per-dataset results show that TPD improves visual prompt baselines under both shifted-domain and cross-dataset evaluation, while also outperforming simpler stabilization approaches. The results distinguish gains from trajectory-level control from instability in naive test-time adaptation of visual prompts.

D.2 Full Per-Dataset Results

  • Per-dataset expansions of the main comparisons and ablations show that mean improvements reflect broad gains rather than a few outlier datasets.Tables A1–A4 expand simple-stabilization comparisons, and Tables A5–A10 expand component ablations.Source: §D.2 · p. 18
  • Naive test-time entropy adaptation of visual prompts can yield an OOD mean below frozen ViT zero-shot, 39.66% versus 46.17%, while TPD surpasses zero-shot under shifted-domain evaluation.With TPD, visual prompts also remain competitive under cross-dataset transfer.Source: §D.2 · pp. 18–19

D.2.1 Simple stabilization baselines.

  • Clipping, norm control, EMA, and low-pass filtering improve raw updates on some datasets, but their gains are smaller and less consistent than TPD's.These alternatives restrict update scale or smooth trajectories without separating persistent, oscillatory, and residual components.Source: §D.2.1 · p. 19

D.2.2 Full detailed ablations.

The detailed ablations compare alternative Koopman estimators, projection choices, and routing rules. Across datasets, the diagonal estimator, Hankel projection, and full routing rule show the most stable overall behavior.

D.2.2 Full detailed ablations.

  • Across datasets, the diagonal Koopman estimator and Hankel projection show more stable overall behavior than the alternatives tested.The ablations compare full versus diagonal local estimators and plain projection versus the Hankel construction.Source: §D.2.2 · p. 20
  • The full HUR routing rule—preserving persistent components while attenuating oscillatory and residual components—shows the most stable overall behavior across datasets.The routing ablation tests whether this selective routing is beneficial.Source: §D.2.2 · p. 20
  • When episodic update-history statistics are reset after every episode, TPD applies identity control and the result is exactly the episodic baseline.The prompt parameters are reset after each episode; TPD normally retains only lightweight update-history statistics across episodes.Source: §D.2.2 · p. 20

D.3 Hyperparameter Sensitivity

Sensitivity analyses report broad high-performing regions rather than isolated parameter settings, with one fixed configuration used across datasets, prompt types, backbones, and baselines. The persistent ratio is also examined as a trajectory-level diagnostic related to adaptation gain.

D.3 Hyperparameter Sensitivity

  • A single fixed TPD configuration is used across datasets, prompt modalities, backbone families, and baselines, without target-specific hyperparameter selection.The sensitivity sweeps are diagnostic robustness analyses rather than per-dataset or per-baseline tuning.Source: §D.3 · p. 22
  • Short-to-moderate Hankel windows form a broad high-performing region across domain generalization and cross-dataset transfer, supporting K=3 and M=8.Very short histories provide insufficient context, while overly long histories may mix outdated updates into the local model.Source: §D.3 · p. 22, Figure 6 · p. 22, Figure 7 · p. 22
  • AKC performs strongly around ρ̄=1.0 with a small gain floor; large gmin weakens shrinkage, while overly strict thresholds can suppress useful updates.The reported default is ρ̄=1.0 and gmin=0.1.Source: §D.3 · p. 23, Figure 8 · p. 23, Figure 9 · p. 23
  • HUR's high-performing routing region favors strong residual suppression and moderate oscillatory attenuation; performance degrades as both routing weights approach one.Persistent components retain unit weight, while oscillatory components are attenuated rather than discarded.Source: §D.3 · p. 23, Figure 10 · p. 23, Figure 11 · p. 24

D.4 Persistent Ratio and Adaptation Gain

  • Across TPT datasets, adaptation gain generally rises from low to moderate persistent ratios, peaks at an intermediate range, then saturates or slightly decreases on several datasets.The persistent ratio measures the fraction of an update aligned with locally consistent trajectory directions.Source: §D.4 · p. 24, Figure 12 · p. 24

D.5 Nonstationary Streams and Test-Order Variation

This section examines TPD across nonstationary streams, random seeds, and magnitude-controller comparisons, including component analyses when update directions are weakly predictable, and reports experimental details, statistical analyses, and code availability. It also states that the work involved no human-subjects or crowdsourcing research and that LLMs were not part of the core method or experiments.

D.5 Nonstationary Streams and Test-Order Variation

  • TPD improves overall accuracy by 1.42, 1.53, and 4.44 percentage points across Abrupt, Mixed, and Rapid Alternation streams, respectively, while reducing order-to-order standard deviation in all three.Under Rapid Alternation, First-10 accuracy rises from 45.82% to 50.09%; this stream switches domains after every adaptation update.Source: §D.5 · p. 25, Table A11 · p. 25

D.6 Component Ratios under Nonstationary Shifts

  • Under Rapid Alternation, VPT’s persistent fraction falls from 17.54% to 5.09% and its residual fraction rises from 81.75% to 93.65%, yet TPD improves accuracy in every stream.TPD has its largest VPT gain under Rapid Alternation, and TPT also benefits despite residual fractions exceeding 96%; the ratios measure local trajectory predictability, not semantic correctness, motivating soft attenuation with nonzero routing weights.Source: §D.6 · p. 25, Table A12 · p. 25

D.7 Variation across Random Seeds

  • Across three random seeds, TPD improves mean accuracy in all four VPT and TPT settings and reduces standard deviation in three; VPT on ImageNet-A rises from 19.25 ± 1.97% to 38.73 ± 1.28%.TPT on ImageNet-A standard deviation increases slightly, from 1.12 to 1.16, so improved mean accuracy does not uniformly reduce run-to-run variation.Source: §D.7 · p. 25, Table A13 · p. 26

D.8 Comparison with Adaptive Magnitude Controllers

  • TPD achieves the highest accuracy in all four settings against Adaptive Clipping and Variance-aware Scaling; on VPT/ImageNet-A it reaches 38.98%, versus 23.52% and 22.76%, respectively.The comparison suggests that the tested magnitude-only controllers do not fully account for gains from joint magnitude control and directional routing.Source: §D.8 · p. 26, Table A14 · p. 26

NeurIPS Paper Checklist

  • The checklist reports that experiments and the appendix specify datasets, baselines, hyperparameters, and protocols, and that the code is available; it also notes extensive analyses supporting robustness across settings.The appendix reports error bars and describes their variability factors and calculation method.Source: NeurIPS Paper Checklist · pp. 28–30
  • The paper marks crowdsourcing and human-subjects research as not applicable.The passage also notes that human-subjects research may require IRB approval, depending on the country and institution.Source: NeurIPS Paper Checklist · p. 32
  • The core method and experiments do not use LLMs, so LLM usage is marked not applicable.Source: NeurIPS Paper Checklist · p. 33

Related papers

Loading 2610.35090v1…