PaperlayerView exact revision on arXiv

Source-linked AI summary

TTRV: Test-Time Reinforcement Learning for Vision Language Models

Akshit Singh, Shyam Marjit, Wei Lin, Paul Gavrikov, Serena Yeung-Levy, Hilde Kuehne, Rogerio Feris, Sivan Doveh, James Glass, M. Jehanzeb Mirza

arXiv:2510.06783v2cs.CV

TL;DR

Vision-language models typically need annotated data for adaptation, unlike learning from unlabeled experience. TTRV applies reinforcement learning at test time using rewards from output frequency and diversity, and reports consistent gains across recognition and VQA, including improvements with a single unlabeled example.

  • Problem

    VLM adaptation typically requires large amounts of annotated data, unlike learning from unlabeled experiences in the world.

    Source: §1 · p. 1

  • Method

    TTRV applies GRPO to unlabeled test data, rewarding frequent outputs while controlling output diversity through a low-entropy reward.

    Source: §1 · p. 2, §3.2 · pp. 4–5

  • Results

    TTRV consistently improves performance across 16 datasets spanning recognition and VQA; gains reach 52.4% when post-training InternVL3 on 20 randomly sampled test images.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors position test-time RL as a potential bridge between pretraining and deployment, with gains reported even from rewards based on one randomly chosen example.

    Source: §1 · p. 2, §5 · p. 9

    The authors do not yet provide a theoretical explanation for why TTRV enhances task-specific abilities rather than adapting to the data distribution.

    Source: §5 · p. 8

Abstract

from arXiv · show

Existing methods for extracting reward signals in Reinforcement Learning typically rely on labeled data and dedicated training splits, a setup that contrasts with how humans learn directly from their environment. In this work, we propose TTRV to enhance vision language understanding by adapting the model on the fly at inference time, without the need for any labeled data. Concretely, we enhance the Group Relative Policy Optimization (GRPO) framework by designing rewards based on the frequency of the base model's output, while inferring on each test sample multiple times. Further, we also propose to control the diversity of the model's output by simultaneously rewarding the model for obtaining low entropy of the output empirical distribution. Our approach delivers consistent gains across both object recognition and visual question answering (VQA), with improvements of up to 52.4% and 29.8%, respectively, and average boosts of 24.6% and 10.0% across 16 datasets. Remarkably, on image recognition, TTRV applied to InternVL 8B surpasses GPT-4o by an average of 2.3% over 8 benchmarks, while remaining highly competitive on VQA, demonstrating that test-time reinforcement learning can match or exceed the strongest proprietary models. Finally, we find many interesting properties of test-time RL for VLMs: for example, even in extremely data-constrained scenarios, where adaptation is performed on a single randomly chosen unlabeled test example, TTRV still yields non-trivial improvements of up to 5.5% in recognition tasks.

1. Introduction

TTRV adapts vision-language models at inference time using rewards derived from unlabeled test data, addressing reliance on labeled data and curated training splits. Evaluation reports gains across recognition and VQA, including cross-dataset and extremely low-data settings.

1. Introduction

  • Existing VLM adaptation typically requires annotated data and costly fine-tuning, while RL methods often use human-labeled rewards and curated training splits.The paper frames this setup as unlike learning from ambiguous, unlabeled experiences in the world.Source: §1 · pp. 1–2
  • TTRV uses Group Relative Policy Optimization with rewards based on output frequency and lower entropy of the empirical output distribution.The approach infers multiple times on each unlabeled test sample and rewards more frequent predictions while controlling output diversity.Source: §1 · p. 2
  • With 20 randomly sampled test images, InternVL3 GRPO gains up to 52.4%, while VQA improves by up to 28.0% on AI2D.The evaluation spans 16 datasets across image recognition and VQA.Source: §1 · p. 2
  • Training on one dataset can yield strong gains on an unrelated dataset, and a single randomly chosen example yields up to 5.5% improvement.The authors suggest these findings reflect activation of latent capabilities from large-scale pretraining rather than simply adapting to a dataset distribution.Source: §1 · p. 2
  • TTRV is presented as the first test-time RL framework for VLMs, with consistent improvements across tasks, model families, and domains.The authors describe it as bootstrappable to any pretrained VLM and requiring no supervised data.Source: §1 · p. 2

2. Related Work

Prior VLM work includes dual encoders for recognition and large multimodal models for open-ended tasks; RL fine-tuning and test-time training offer distinct adaptation approaches. TTRV applies test-time RL to decoder-based VLMs using frequency rewards with entropy regularization.

Related Work

  • VLMs include dual-encoder models suited to recognition and large multimodal models supporting open-ended tasks such as VQA and document understanding.Examples include CLIP [46] and ALIGN [21] among dual encoders, and BLIP-2 [25] and the LLaVA series [24] [30] [31] among large multimodal models.Source: §2 · p. 2
  • Most test-time training approaches use entropy minimization or self-supervised losses; TPT [55] tunes text prompts using entropy over augmented views.Other approaches include DiffTPT [15] and C-TPT [68], which improve augmentation quality and calibration, and RLCF [74], which adapts the image encoder using larger-model feedback.Source: §2 · p. 3
  • RL fine-tuning methods such as VLM-R1 [54], VisualThinker-R1-Zero [75], and Perception-R1 [69] target vision-driven tasks but rely on curated training splits or labeled feedback.Prior work also includes RLHF [43], DPO [47], and rule-based GRPO [53].Source: §2 · p. 3
  • Many test-time training methods have constraints that limit decoder-based VLMs: Norm [49] and DUA [35] require batch normalization, while TENT [60] uses class-probability entropy.Decoder-based VLMs generate autoregressive token distributions rather than class-level distributions, making direct TENT-style objectives nontrivial.Source: §2 · p. 3
  • TTRV extends test-time RL to decoder-based VLMs, combining frequency-based rewards with entropy regularization rather than relying on majority voting alone.The related method TTRL [78] uses majority voting across sampled outputs as a surrogate reward for test-time RL in LLMs.Source: §2 · p. 3

3. TTRV: Test Time RL for VLMs

TTRV applies GRPO at inference using self-supervised rewards derived from unlabeled test-sample output distributions. Its frequency and entropy terms combine repeated-answer consistency with diversity control.

3.1. Recap: Group Relative Policy Optimization

  • GRPO optimizes sampled responses relative to one another, using clipped importance-weighted updates and KL regularization to constrain deviation from a reference policy.The KL-regularized objective uses β to control regularization strength.Source: §3.1 · p. 4

3.2. Test-Time RL with Distributional Rewards

  • The frequency reward gives more credit to repeated outputs while retaining nonzero reward for alternatives, unlike best-of-N selection of only the most frequent response.It estimates output frequencies across multiple samples from the current policy for each test example.Source: §3.2 · pp. 4–5
  • An entropy-derived auxiliary term penalizes dispersion in the output distribution, while α tunes the trade-off between convergence and diversity.The combined reward adds the frequency-based and entropy terms.Source: §3.2 · p. 5

3.3. Optimization Objective

  • For decoder-based VLMs, the reward provides soft sample-level weights for autoregressive token learning, while GRPO uses relative advantages for more stable updates.GRPO replaces raw rewards with comparisons within the sampled group.Source: §3.3 · p. 5

4. Results

TTRV improves InternVL image recognition and VQA results across diverse benchmarks, and ablations report gains under cross-dataset transfer and data-scarce settings. The experiments also compare reward designs, sampling strategies, and model families.

4.1. Evaluation Settings

  • Evaluation covers eight image-recognition benchmarks and eight VQA datasets, including fine-grained recognition, remote sensing, math reasoning, compositional reasoning, and chart questions.Capture is included as an open-ended VQA dataset to assess performance on open-ended generative tasks.Source: §4.1 · pp. 5–6

4.2. Results

  • Around 32.9% average gain across image-recognition datasets for InternVL3-2B; TTRV also lifts InternVL3-8B above 99% on ImageNet, outperforming proprietary systems including GPT-4o.The reported gains use 20 randomly sampled test instances per dataset; the authors note some cases of decreased performance, including Resisc45 with InternVL2.5-4B.Source: Table 1 · p. 6, §4.2 · p. 7
  • TTRV consistently improves VQA results across datasets and model scales; InternVL3-8B gains 12.4% on CRPE and 7.5% on RealWorldQA.TTRV is reported to lag GPT-4o by approximately 2% on average while outperforming it on some benchmarks, including MathVista.Source: Table 2 · p. 6, §4.2 · p. 7

4.3. Ablations

  • The combined frequency-based and diversity-control rewards outperform the alternative reward designs, including majority voting and entropy minimization.The paper describes TTRV without the frequency reward as test-time entropy minimization proposed by TENT.Source: §4.3 · p. 7, Table 3 · p. 8
  • TTRV shows strong cross-data generalization: applying it on one dataset yields gains when testing on a completely different distribution.The authors interpret this as enhancement of underlying task ability rather than distribution-specific adaptation.Source: §4.3 · p. 7, Figure 3 · p. 8
  • Biased sampling from only a subset of classes still yields substantial improvements over the base model, and measurable gains remain with a single randomly chosen test example.The biased-sampling comparison includes sampling from 4 of 200 ImageNet-R classes; the single-example setting covers image classification and VQA.Source: §4.3 · pp. 7–8, Table 4 · p. 8, Table 6 · p. 8
  • InternVL models do not appear to benefit from random rewards, suggesting their TTRV gains stem from meaningful reward signals rather than spurious correlations.The comparison follows work reporting gains from spurious rewards in some GRPO-trained models.Source: §4.3 · p. 7, Table 5 · p. 8
  • TTRV also yields consistent gains with Qwen2.5-VL, extending the reported results beyond the InternVL model family.The ablation reports results for image classification and VQA using Qwen2.5-VL-3B.Source: Table 7 · p. 8, §4.3 · p. 8

5. Limitations and Conclusion

TTRV uses frequency- and diversity-based rewards to adapt VLMs on unlabeled test data, with reported gains across 16 benchmarks. The authors also note that the behavior lacks a theoretical explanation.

5. Limitations and Conclusion

  • The method combines a reward based on prediction frequency with a second reward that regulates diversity in rollouts.The conclusion describes both rewards as extracted on the fly from unlabeled test data.Source: §5 · p. 8
  • TTRV enhances task-specific abilities of the base model, but the authors do not yet provide a theoretical explanation for this behavior.They identify establishing such a foundation as future work.Source: §5 · pp. 8–9
  • TTRV consistently improves performance across 16 benchmarks spanning object recognition and visual question answering, even outperforming GPT-4.The conclusion also reports gains over strong base models.Source: §5 · pp. 8–9
  • The authors report that ablations reveal data-efficient properties and task-specific improvements without explicit supervision.Source: §5 · p. 9

Appendix

The supplement provides additional experiments, explanations, implementation and evaluation details, dataset descriptions, prompts, ablations, and pseudocode.

Appendix

  • The supplement includes additional experiments and explanations intended to provide further insight beyond the main manuscript.Source: Appendix · p. 12
  • Supplementary sections describe implementation details, evaluation protocols, datasets, and experimental prompts.Source: Appendix · p. 12
  • The supplement includes further ablations and comprehensive pseudocode to clarify implementation and support reproducibility.Source: Appendix · p. 12
  • Experiments used four NVIDIA A100 and four NVIDIA A6000 GPUs, and the authors provide code and execution instructions for review.They state that the codebase will be released publicly upon acceptance.Source: Appendix · p. 12

6. Additional Experimental Settings

The additional experimental settings specify optimization and rollout parameters, adaptation sample counts, and task-specific evaluation procedures.

6. Additional Experimental Settings

  • Optimization uses AdamW with cosine scheduling and a peak learning rate of 5 × 10−7; rollouts generate 32 responses at temperature 1.0, with α fixed at 0.75.Prompt and response lengths are capped at 7524 and 1024 tokens, respectively.Source: §6 · p. 12
  • Main-table results generally use 20 randomly sampled test examples for adaptation, with comparisons to 500 samples and an ablation using one example.Source: §6 · p. 12
  • Evaluation uses greedy decoding, multiple-choice formats for recognition and most VQA tasks, and accuracy except for Capture, which uses 1 −symmetric mean percentage error.Capture uses free-form answers, while MME yes/no questions are converted to multiple choice; zero-shot results omit chain-of-thought prompting.Source: §6 · p. 12

7. Dataset Description

The evaluation dataset collection spans recognition and VQA challenges, including distribution shifts, fine-grained recognition, mathematical reasoning, compositionality, and real-world understanding.

7. Dataset Description

  • The study curates recognition and VQA benchmarks, with dataset statistics reporting original test sizes and post-preprocessing image counts.Source: §7 · p. 12
  • Recognition datasets cover standard and adversarial object recognition, sketch and texture shifts, fine-grained categories, and material recognition.The collection includes ImageNet, ImageNet-V2, ImageNet-A, ImageNet-Sketch, ImageNet-R, Food101, and DTD.Source: §7 · p. 12
  • VQA datasets probe mathematical reasoning, general multimodal understanding, real-world scenarios, counterfactual reasoning, compositionality, hallucination resistance, and diagram understanding.The collection includes Mathverse, MathVista, Seed, MME, RealWorldQA, Capture, CRPE, and AI2D.Source: §7 · p. 12
  • Images exceeding 1000 × 1000 pixels were filtered out, while RealWorldQA images were explicitly resized to that maximum resolution.Source: §7 · p. 12

8. TTRV Prompt Details

The appendix gives representative prompts for each dataset, noting that prompts vary especially for VQA. Examples use image questions and constrained answer formats, while dataset statistics account for an image-resolution filter.

8. TTRV Prompt Details

  • For each dataset, the paper presents a representative prompt example; specific prompts may vary with the question, particularly in VQA, so it provides a general outline of their structure and format.Source: §8 · pp. 12–13
  • Dataset statistics distinguish original test-image counts from counts used after excluding images above 1000 × 1000 resolution.Table 8 reports both counts.Source: Table 8 · p. 13
  • Recognition prompts ask what an image shows and instruct the model to answer with only the corresponding option letter.Examples include prompts for ImageNet and ImageNet-R.Source: §8 · p. 13
  • VQA examples include direct questions and multiple-choice formats, with prompts requiring an option letter and no extra text.Examples include a yes-or-no home-office question and a question about a wind turbine.Source: §8 · p. 14

9. Additional Experiments

Additional experiments report cross-dataset gains, improvements on additional model families, and robustness under greedy decoding. They also describe the performance–latency trade-off as adaptation uses more samples.

9.1. Latency versus Number of Samples

  • Adaptation with 500 rather than 20 samples consistently improves performance, but larger adaptation sets increase memory use, processing time, and inference latency.The experiments compare performance across sample sizes and separately measure the associated latency.Source: §9.1 · p. 15

9.2. Robustness

  • With greedy decoding, TTRV shows strong robustness, with only minor performance variations attributed to hardware and software factors.Results are summarized across five independent runs.Source: §9.2 · p. 15

9.3. Further Cross-Data Generalization Examples

  • Training on ImageNet-V2 yields +15.89% on ImageNet-R and +5.63% on ImageNet-A; training on Food and testing on MathVista yields +2.52%.The reported cross-dataset results include transfers across recognition datasets and from visual recognition to VQA.Source: §9.3 · p. 15

9.4. Additional Models

  • Additional experiments report consistent improvements on MM-Eureka, ThinkLite-VL, and VisionReasoner.The authors present these results as further evidence that the method is model-agnostic.Source: §9.4 · p. 15

10. Pseudocode

The pseudocode adapts a decoder-based VLM on a test sample through repeated rollouts, empirical-frequency rewards with entropy control, and a GRPO policy update. It returns the adapted model.

10. Pseudocode

  • The procedure returns the model with adapted parameters after updating the decoder-based VLM on the test sample.The pseudocode lists the learning rate as an input to the policy update.Source: §10 · p. 17
  • For each test sample, the method generates N responses and derives empirical response probabilities from their frequencies.The pseudocode gives N=32 as its default rollout count.Source: §10 · p. 17
  • Each response receives a frequency-based reward, combined with a shared negative-entropy reward weighted by alpha.The pseudocode sets the entropy reward to -H and uses alpha to weight it.Source: §10 · p. 17
  • The rewards are converted to relative advantages using their mean and standard deviation before the GRPO policy update.A small-standard-deviation check avoids division by zero.Source: §10 · p. 17

Related papers

Loading 2510.06783v2…