PaperlayerView exact revision on arXiv

Source-linked AI summary

Test-Time Logit Prompting for Source-Free Missing Modality Adaptation

Taixi Chen, Nancy Guo

arXiv:2609.02039v1cs.CV

TL;DR

Missing-modality inputs degrade VLM recognition, while existing robustness methods rely on inaccessible source training data. TLP adapts frozen VLMs at test time by optimizing missing-aware logit prompts with uncertainty-aware adjustment and modality-complete consistency regularization. Across vision-language benchmarks, it consistently improves missing-modality robustness, with gains up to 8% using hundreds of tunable parameters and few optimization steps.

  • Problem

    Missing-modality inputs can degrade VLM performance, and existing compensation strategies depend on source training data that may be unavailable.

  • Method

    TLP performs source-free test-time adaptation by optimizing missing-aware logit prompts for frozen VLMs with uncertainty-aware adjustment and modality-complete consistency regularization.

  • Results

    TLP consistently enhances missing-modality robustness across multiple vision-language benchmarks, achieving improvements of up to 8%.

  • Takeaways & Limitations

    TLP provides efficient missing-modality adaptation using only a few optimization steps and hundreds of tunable parameters.

  • Takeaways & Limitations

    Adaptation assumes the original training dataset is inaccessible and only unlabeled test samples are available.

Abstract

from arXiv · show

Vision-language models (VLMs) have achieved remarkable performance by leveraging complementary information from large-scale image-text pairs. However, missing-modality inputs are commonly encountered during real-world deployment, often leading to significant performance degradation. Existing methods primarily enhance model robustness by learning modality compensation strategies from source training data. However, their reliance on source training data makes them difficult to apply when original data are unavailable due to privacy, storage, or accessibility constraints, such as clinical applications and personalized AI services. This raises an important yet underexplored question: can VLMs be efficiently adapted at test time for visual recognition with missing modalities without accessing source training data? To this end, we propose Test-Time Logit Prompting (TLP), a lightweight source-free test-time adaptation framework for visual recognition with missing modalities. To address missing-induced prediction shifts, TLP optimizes logit prompts with uncertainty-aware adjustment and modality-complete consistency regularization, adaptively adjusting prediction confidence while preserving semantic consistency. Extensive experiments across diverse vision-language benchmarks demonstrate that TLP consistently enhances recognition performance under missing-modality scenarios, achieving up to 8\% improvements while requiring only hundreds of tunable parameters and a few test-time optimization steps.

Introduction

Missing modalities disrupt VLM cross-modal alignment, while existing robustness strategies depend on source training data. TLP addresses this gap through lightweight source-free test-time adaptation in the logit space.

  • Real-world missing-modality inputs can result from privacy constraints, sensor failures, or incomplete data collection.
  • Existing methods improve robustness through cross-modal generation, joint representation learning, or prompt learning using source training data.
  • TLP asks whether VLMs can improve missing-modality robustness at test time without accessing source training data.
  • TLP adapts frozen VLMs by optimizing lightweight missing-aware logit prompts under different missing-modality conditions.
  • Uncertainty-aware adjustment and modality-complete consistency regularization address confidence bias and semantic inconsistency during logit-level adaptation.
  • Experiments report consistent improvements across diverse missing-modality scenarios using only hundreds of parameters and test-time optimization.

Related Works

Missing-modality learning remains challenging because incomplete information disrupts cross-modal alignment. Existing approaches include cross-modal generation, joint representation learning, and prompt learning, while test-time adaptation typically targets modality-complete distribution shifts.

  • Missing-modality learning remains challenging because incomplete multimodal information disrupts cross-modal alignment.
  • Existing missing-modality methods include cross-modal generation, joint representation learning, and prompt learning.
  • Test-time adaptation improves performance during inference without accessing source training data.
  • Existing test-time adaptation methods update normalization statistics, optimize model parameters, or adapt lightweight modules.

Proposed Method

TLP formulates source-free test-time adaptation for missing-modality recognition using frozen VLMs and unlabeled incomplete test samples. It adjusts predictions with modality-aware logit prompts, uncertainty control, and consistency regularization anchored by complete-modality samples.

  • Problem Formulation: TLP adapts a source-trained VLM using only unlabeled test samples, without accessing original training data or test labels.
  • Problem Formulation: The framework represents missing modalities with dummy inputs, allowing incomplete samples to retain the VLM's expected input format.
  • Test-Time Logit Prompting: TLP keeps the VLM backbone frozen and optimizes lightweight, modality-aware prompts directly in the logit decision space during inference.
  • Test-Time Logit Prompting: A prompt pool provides separate class-wise parameters for complete, m1-available, and m2-available conditions, selected according to each sample's modality availability.
  • Source-Free Prompt Optimization: Uncertainty-aware adjustment reduces over-confident predictions by encouraging adjusted probabilities toward a uniform distribution over classes.
  • Source-Free Prompt Optimization: Modality-complete consistency regularization preserves semantic structure by matching missing-modality predictions to nearest complete-modality anchors, while α = 1/C limits unreliable-anchor influence.
  • Efficiency: Because only class-level logit prompts are trainable, TLP uses |P|×C parameters and incurs minimal computational overhead.

Experiment

Experiments evaluate TLP across three vision-language benchmarks, missing-modality settings, changing missing rates, plug-and-play combinations, ablations, and adaptation efficiency. TLP consistently improves recognition while adapting frozen VLMs with lightweight logit prompts and limited test-time computation.

  • Datasets: Experiments use MM-IMDb, UPMC Food-101, and Hateful Memes to evaluate missing-modality recognition across diverse multimodal classification settings.MM-IMDb is multi-label movie genre classification; Food-101 covers 101 food categories; Hateful Memes requires both visual and textual information.
  • Experimental settings: TLP evaluates source-trained VLMs under missing-text, missing-image, and mixed missing conditions, with unavailable modalities replaced by dummy inputs.The missing rate η denotes the proportion of samples with incomplete modalities.
  • Main results: Up to 8% improvement over the non-adapted baseline is achieved after five optimization steps, while one step already improves performance from 44.76% to 46.91%.One-step adaptation requires 3-90 seconds across datasets.
  • Generalization: TLP consistently improves performance across missing-text, missing-image, and mixed missing rates from 10% to 90% without source data or model-parameter updates.Missing text causes more severe degradation on MM-IMDb, yet TLP maintains gains as the missing rate increases.
  • Plug-and-play evaluation: TLP further improves existing training-based missing-modality methods across scenarios and missing rates, with gains remaining stable or becoming more pronounced as missingness increases.This supports TLP's use as a plug-and-play test-time adaptation component.
  • Ablation analysis: Removing both objectives lowers performance, while either uncertainty-aware adjustment or modality-complete consistency improves results; jointly optimizing them achieves the best performance across benchmarks.The two objectives provide complementary guidance for source-free logit prompt adaptation.

Conclusion

The paper presents TLP, a source-free test-time adaptation framework that improves frozen VLMs for missing-modality recognition by optimizing missing-aware logit prompts. Experiments show consistent robustness gains with few optimization steps and hundreds of tunable parameters.

  • TLP adapts frozen VLMs for missing-modality recognition by optimizing missing-aware prompts directly in logit space without source training data.
  • Uncertainty-aware adjustment and modality-complete consistency regularization support prediction adjustment while preserving semantic information.
  • TLP consistently enhances missing-modality robustness across multiple vision-language benchmarks with only a few optimization steps and hundreds of tunable parameters.
  • TLP can be integrated with existing training-based methods as an efficient plug-and-play framework for incomplete multimodal scenarios.
Loading 2609.02039v1…