PaperlayerView exact revision on arXiv

Source-linked AI summary

Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuning

Chun-Mei Feng, Kai Yu, Yong Liu, Salman Khan, Wangmeng Zuo

arXiv:2308.06038v2cs.CV

TL;DR

Test-time prompt tuning must adapt to unseen domains despite limited diversity from conventional augmentation and insufficient prediction fidelity from entropy-based selection. DiffTPT combines conventional and Stable Diffusion augmentation with cosine similarity filtration, improving zero-shot accuracy by an average of 5.13% over state-of-the-art TPT.

  • Problem

    Test-time prompt tuning needs to adapt prompts from a single unlabeled sample in an unseen domain, while existing augmentation lacks diversity and entropy selection may not ensure prediction fidelity.

  • Method

    DiffTPT combines conventional and Stable Diffusion augmentation, using the test sample’s CLIP image feature and cosine similarity filtration to select generated images.

  • Results

    5.13% average zero-shot accuracy improvement is reported over the state-of-the-art TPT method.

  • Takeaways & Limitations

    DiffTPT balances richer visual variation with preserved key semantics and prediction fidelity for test-time prompt tuning.

  • Takeaways & Limitations

    Diffusion-based augmentation can introduce spurious images that lower data fidelity and collapse prompt-tuning performance.

Abstract

from arXiv · show

Benefiting from prompt tuning, recent years have witnessed the promising performance of pre-trained vision-language models, e.g., CLIP, on versatile downstream tasks. In this paper, we focus on a particular setting of learning adaptive prompts on the fly for each test sample from an unseen new domain, which is known as test-time prompt tuning (TPT). Existing TPT methods typically rely on data augmentation and confidence selection. However, conventional data augmentation techniques, e.g., random resized crops, suffers from the lack of data diversity, while entropy-based confidence selection alone is not sufficient to guarantee prediction fidelity. To address these issues, we propose a novel TPT method, named DiffTPT, which leverages pre-trained diffusion models to generate diverse and informative new data. Specifically, we incorporate augmented data by both conventional method and pre-trained stable diffusion to exploit their respective merits, improving the models ability to adapt to unknown new test data. Moreover, to ensure the prediction fidelity of generated data, we introduce a cosine similarity-based filtration technique to select the generated data with higher similarity to the single test sample. Our experiments on test datasets with distribution shifts and unseen categories demonstrate that DiffTPT improves the zero-shot accuracy by an average of 5.13\% compared to the state-of-the-art TPT method. Our code and models will be publicly released.

1. Introduction

Test-time prompt tuning adapts prompts for each unlabeled sample from an unseen domain, but existing augmentation and confidence selection limit diversity and prediction fidelity. DiffTPT combines diffusion-based augmentation with cosine similarity filtration to balance these needs and improves zero-shot accuracy over prior TPT.

  • Test-time prompt tuning learns adaptive prompts for individual samples from unseen domains without training data or annotations.
  • Existing TPT uses simple transformations that provide insufficiently diverse augmented views, potentially causing prompt overfitting.
  • Diffusion models offer higher-diversity augmented images with richer visual variation and can preserve key semantics.
  • DiffTPT combines conventional and diffusion-based augmentation with cosine similarity filtration to balance data diversity and prediction fidelity.
  • 5.13% average zero-shot accuracy improvement is reported over the state-of-the-art TPT method.

2. Related Work

Related work spans prompt tuning, test-time optimization, and image synthesis. DiffTPT extends test-time prompt tuning by using diffusion-generated images to provide diverse information while retaining the test sample’s semantics.

  • Prompt Tuning: Prompt tuning adapts large pre-trained models to downstream tasks through parameter-efficient learned prompts.
  • Test-time Optimization: Prior TPT generates random augmented views of one test sample and filters noisy augmentations through entropy minimization.
  • Test-time Optimization: Existing test-time optimization methods address inference without training data but face limitations involving required test-sample counts or architecture scalability.
  • Image Synthesis: Standard image manipulations use predefined transformations, whereas image synthesis offers more flexible and diverse content augmentation.
  • Image Synthesis: DiffTPT directly augments test data with semantically consistent but diverse diffusion-generated information for test-time prompt tuning.

3. Methodology

DiffTPT adapts CLIP prompts at test time by combining conventional and diffusion-based augmentations, then filtering synthetic views to balance diversity with prediction fidelity.

  • Test-time Prompt Tuning: Test-time prompt tuning optimizes learnable class prompts using only a single unseen-domain test sample and no labeled training data.
  • Test-time Prompt Tuning: Existing TPT generates multiple random augmented views and uses entropy-based confidence selection, but simplistic transformations can limit diversity and cause prompt overfitting.
  • Diffusion-based Diverse Data Augmentation: DiffTPT uses Stable Diffusion with the test image's CLIP feature to generate augmented images with richer visual appearance variation while preserving key semantics.
  • Diffusion-based Diverse Data Augmentation: DiffTPT combines conventional and diffusion-based augmentations to exploit their complementary merits during adaptive prompt learning.
  • Filtration with Cosine Similarity: Cosine similarity filtration retains augmented images whose similarity to the test sample exceeds ε, removing spurious views that may reduce data fidelity.
  • Filtration with Cosine Similarity: The resulting objective selects views using both entropy and cosine-similarity masks, providing many appearance-varied samples while preserving key semantics for prompt optimization.

4. Experiments

DiffTPT is evaluated under natural distribution shifts and cross-dataset generalization, where it consistently outperforms competing methods and supports ablations of its augmentation and prompt-update choices.

  • Comparison with State-of-the-arts: DiffTPT outperforms competing methods on five natural-distribution-shift datasets under Scenario 1.The evaluation uses ResNet-50 and ViT-B/16 backbones, with ImageNet and OOD datasets.
  • Comparison with State-of-the-arts: 5.1% and 2.3% performance gains against TPT are reported for ResNet-50 and ViT-B/16, respectively, under cross-dataset generalization.Average accuracy increases from 55.12 to 59.85 and from 63.03 to 65.47 across the evaluated settings.
  • Ablation Studies: Higher ρC gradually improves accuracy, whereas ρH values above 0.5 decrease accuracy.The paper associates ρC with cosine-similarity filtering of diffusion augmentations and ρH with entropy filtering of standard views.
  • Ablation Studies: Accuracy rises with more augmented views until a plateau around N = 64, while N = 8 still yields gains exceeding 3.5% and 4.6%.These gains are reported for zero-shot CLIP in the two scenarios; N = 128 does not degrade accuracy.
  • Ablation Studies: 58.8 accuracy is reached on S2 by increasing prompt-update steps from 1 to 4, but performance decreases slightly beyond 5 steps.The method uses four optimization steps by default because additional updates also increase inference time.

5. Conclusion

DiffTPT combines diffusion-based augmentation with cosine similarity filtration for test-time prompt tuning, producing diverse yet semantically consistent samples while preserving prediction fidelity. Across zero-shot generalization tasks, it improves accuracy over TPT by an average of 5.13%.

  • DiffTPT combines diffusion-based augmentation with cosine similarity-based filtration.The diffusion model generates diverse, semantically consistent images, while filtration enforces prediction fidelity.
  • 5.13% average zero-shot accuracy improvement over the state-of-the-art TPT method.The improvement is reported across various zero-shot generalization tasks.

A. Comparisons of the Full Dataset on S1 and S2

On the full S1 and S2 datasets, DiffTPT outperforms competing methods across nearly all datasets and yields stronger average improvements than TPT for both evaluated backbones.

  • DiffTPT outperforms other methods on almost all datasets in the full-dataset evaluation.The comparison includes ResNet-50 and ViT-B/16 backbones under S1 and S2.
  • 55.87% ImageNet-V2 accuracy for CLIP-RN50 with DiffTPT, versus 51.41% for CLIP-RN50.
  • 41.70% EuroSAT accuracy with ResNet-50 using DiffTPT, versus 22.56% with TPT.
  • 45.20% EuroSAT accuracy with ViT-B/16 using DiffTPT, versus 38.26% with TPT.

B. Proportion Analysis of the Different Augmented Images

The proportion of diffusion-based augmentation affects accuracy non-monotonically: performance improves initially, then declines beyond an optimal mixture, especially under S1.

  • S1 accuracy increases as diffusion-based augmentation grows, then declines when its ratio reaches 0.4.
  • S2 classification performance similarly rises and then falls as the diffusion-based augmentation proportion increases.
  • The more prominent S1 variation supports combining standard and diffusion-based augmentation to exploit complementary advantages.

C. Visualization of the Generated and Filtered Image

The visualization distinguishes original, confidence-filtered, cosine-filtered, jointly filtered, and final diffusion-augmented images. It shows that cosine similarity removes spurious samples while the final set retains diversity and key semantics.

  • The figure uses gray, orange, green, blue, and red boxes for original, entropy-filtered, cosine-filtered, jointly filtered, and final diffusion-augmented images.
  • Final diffusion-based augmentation provides many diverse and informative images while preserving the same key semantics.
  • Cosine similarity filtering removes augmentations that deviate from the main target content.
  • Entropy-based confidence selection deletes additional spurious augmentations remaining after cosine filtering.
  • Figure A1 plots Top 1 accuracy against the proportions of standard and diffusion-based augmented views under S1 and S2.
Loading 2308.06038v2…