PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Test-Time Scaling for Small Vision-Language Models

Mehmet Onurcan Kaya, Desmond Elliott, Dim P. Papadopoulos

arXiv:2510.03574v2cs.LGcs.CV

TL;DR

Small VLMs can be efficient but have limited generalization, while existing test-time scaling methods can be computationally demanding. The paper proposes augmentation and adaptation strategies using model-internal signals, and reports consistent gains across benchmarks with minimal overhead.

  • Problem

    Small VLMs can have weaker generalization under domain shift, while existing test-time scaling methods may rely on external models or costly computation that challenges resource-constrained use.

    Source: §1 · p. 1

  • Method

    The paper combines token-level aggregation over augmented inputs without parameter updates (TTAug) with inference-time parameter adaptation using TTAug consensus pseudolabels (TTAdapt).

    Source: §1 · pp. 1–2

  • Results

    Both methods consistently improve performance and outperform existing test-time scaling approaches with minimal overhead; TTAug reports a +4.1% absolute improvement over the baseline model.

    Source: §4.1 · p. 5, §5 · p. 10

  • Takeaways & Limitations

    The methods are suitable for resource-constrained environments, and improvements transfer across diverse models, though transferability varies by family and size.

    Source: §5 · p. 10, §4.8 · p. 10

    TTAdapt can occasionally degrade performance on specialized benchmarks with strong baselines, so adaptation intensity should vary by task.

    Source: §4.7 · p. 10

Abstract

from arXiv · show

Small Vision-Language Models (VLMs) provide a computationally efficient alternative to larger models, at the cost of weaker generalization abilities and downstream task performance. These shortcomings could be addressed by test-time scaling techniques, but existing methods are typically computationally demanding, contradicting the resource-efficient design goals of small models. To address these limitations, we propose two novel and efficient test-time scaling strategies that leverage the model-internal features rather than external supervision: (i) Test-Time Augmentation (TTAug), which generates multiple augmented inputs and aggregates outputs at the token level without parameter updates, and (ii) Test-Time Adaptation (TTAdapt), which adapts model parameters during inference using consensus-based pseudolabels from TTAug. Through extensive experiments across nine benchmarks, we demonstrate consistent performance improvements while maintaining computational efficiency suitable for resource-constrained environments. The generality of our approach is demonstrated both within models at different scales and across different VLMs without additional tuning.

1 INTRODUCTION

Small VLMs are efficient but can struggle under domain shift, while existing test-time scaling can be too costly or limited in how it aggregates responses. The paper introduces augmentation and adaptation strategies using model-internal signals and reports consistent improvements over existing methods.

1 INTRODUCTION

  • Many test-time scaling methods depend on external verifiers or costly reranking, while answer-level aggregation can miss token-level quality signals and restrict tasks to extractable answers.Generating complete responses before evaluation can also prevent early stopping and waste computation.Source: §1 · p. 1
  • TTAug creates augmented image-text inputs and aggregates model outputs at the token level without updating parameters.Token-level signals can help detect low-quality responses and use model-internal information more finely.Source: §1 · pp. 1–2
  • TTAdapt fine-tunes model parameters during inference using consensus signals from TTAug as pseudolabels, without labeled data.The paper describes this as a lightweight way to adjust to domain-specific characteristics while retaining computational efficiency.Source: §1 · p. 2
  • The approach consistently outperforms existing test-time scaling methods while keeping computational costs suitable for resource-constrained settings.The paper also reports that greedy decoding with input augmentations is more effective than temperature sampling, and token-level aggregation provides stronger signals than final-answer aggregation.Source: §1 · p. 2
  • The paper presents efficient methods deployable on consumer GPUs, a comprehensive analysis of VLM TTAug, and a multimodal-language-model TTAdapt method.It distinguishes the latter from prior VLM adaptation work focused primarily on CLIP-based models.Source: §1 · p. 2

2 RELATED WORK

Prior test-time scaling, augmentation, and adaptation work leaves practical or coverage gaps for small VLMs. The paper positions its methods as lightweight, systematic approaches that use model-internal signals and consensus pseudolabeling.

2 RELATED WORK

  • Parallel-sampling methods can be impractical for resource-constrained deployments because they often rely on external verifiers or compute-heavy strategies.This conflicts with the resource-efficient design goals of small models.Source: §2 · pp. 2–3
  • Prior VLM TTAug work addressed hallucination detection and robustness more than accuracy improvement, while learnable augmentation policies typically required labeled data.This paper evaluates augmentation design, aggregation, and scaling for VLMs and uses self-supervised objectives to avoid labeled data.Source: §2 · p. 3
  • Most prior multimodal TTAdapt work focused on CLIP-based VLMs, while related language-model and benchmark-specific approaches had limitations in data requirements or broader applicability.The paper presents its source-free method as using consensus-based pseudolabels from TTAug.Source: §2 · p. 3

3 METHODS

The framework combines augmentation-based token-level aggregation with inference-time parameter adaptation guided by consensus pseudolabels. TTAug can be used without parameter updates, while TTAdapt adds iterative optimization and resets weights between questions.

3 METHODS

  • The framework pairs TTAug and TTAdapt to improve small VLM performance and robustness without additional training data or substantial computational overhead.It is presented as a unified framework for test-time scaling.Source: §3 · p. 3

3.1 TEST-TIME AUGMENTATION (TTAUG)

  • TTAug generates multiple semantically equivalent image-text inputs through controlled visual and textual transformations.The transformations preserve content essential for multimodal understanding while introducing input diversity.Source: §3.1 · p. 3
  • At each generation step, TTAug averages next-token distributions across augmented inputs and greedily selects a token for one shared output sequence.The shared context is updated with each selected token, so each later decision uses local confidence signals from the augmented views.Source: §3.1 · pp. 3–4

3.2 TEST-TIME ADAPTATION (TTADAPT)

  • TTAdapt iteratively generates high-confidence pseudolabels from TTAug consensus and fine-tunes model parameters without labeled data.Efficient training can use gradient checkpointing or parameter-efficient methods.Source: §3.2 · p. 4
  • For each question, TTAdapt starts from initial weights, repeats pseudolabel generation and parameter updates, produces an adapted response, then resets the weights.The framework presents TTAug as immediate improvement without parameter updates and TTAdapt for settings where brief optimization is feasible.Source: §3.2 · p. 4

4 EXPERIMENTS

Experiments evaluate the framework on nine benchmarks using 1,000 samples per benchmark, comparing test-time scaling methods and examining design choices, adaptation, and cross-model transfer. Across the studies, token-level TTAug and consensus-guided parameter adaptation show gains, with practical settings and task-dependent limits also reported.

4.1 COMPARISON WITH OTHER TEST-TIME SCALING METHODS

  • +4.1% absolute improvement over the baseline; TTAug outperforms existing test-time scaling methods while using less runtime and generating fewer output tokens.The comparison uses eight candidates for other methods and eight input augmentations for TTAug.Source: Table 1 · p. 5, §4.1 · p. 5

4.2 DIVERSITY-INDUCEMENT METHODS

  • Input perturbations with greedy decoding outperform temperature sampling under both self-consistency and self-selector aggregation, with the largest gains on OCRVQA and GQA.The paper attributes this advantage to stronger alignment with the model training objective.Source: §4.2 · p. 6, Table 2 · p. 6

4.3 AGGREGATION LEVELS

  • Token-level aggregation outperforms answer-level methods, preserving local confidence signals and improving results on OCRVQA, GQA, and COCO, where the baseline struggles.The experiments use simple averaging at the final logits for subsequent studies.Source: Table 3 · p. 6, §4.3 · pp. 6–7

4.4 NUMBER OF AUGMENTATIONS

  • Performance peaks at 16 augmentations on average, although benchmark curves vary and some peak at intermediate counts before declining.At 16 augmentations, inference uses 8.75 GB peak GPU memory and takes 4.77 s per query on an NVIDIA A100 GPU, versus 4.60 GB and 1.43 s at baseline.Source: §4.4 · p. 7, Figure 2 · p. 7

4.5 TEXT AUGMENTATION METHODS

  • Self-paraphrasing performs best overall, while classical text augmentations remain competitive at minimal computational overhead; consistency enforcement is critical to reliable performance.The study uses classical augmentation with consistency enforcement in subsequent experiments to balance accuracy and efficiency.Source: §4.5 · p. 8, Table 4 · p. 8

4.6 IMAGE AUGMENTATION METHODS

  • Classical image augmentations at high and low strengths marginally outperform medium strength; AugMix and generative methods fall short of classical methods.Generative augmentation excludes text-containing images to avoid OCR corruption, reducing coverage; it also requires external diffusion models.Source: §4.6 · p. 9

4.7 TEST-TIME ADAPTATION METHODS

  • Model parameter adaptation yields the strongest overall gains, while aggregation-weight optimization performs around TTAug and can improve confidence-sensitive benchmarks such as AMBER and TextVQA.Performance can degrade on specialized benchmarks with strong baseline capabilities, so the authors suggest task-dependent adaptation intensity.Source: §4.7 · pp. 9–10, Table 6 · p. 10

4.8 CROSS-MODEL GENERALIZATION

  • The framework improves performance across model families and scales, but hyperparameter transfer varies; no universal setting optimally serves all models.The paper reports stronger transfer among models with similar families or parameter counts and recommends dedicated tuning for best results.Source: §4.8 · p. 10, Figure 3 · p. 10

5 CONCLUSION

The paper proposes TTAug and TTAdapt as efficient test-time scaling methods that improve small VLM performance with minimal overhead, while noting that optimal strategies depend on task and model.

5 CONCLUSION

  • TTAug and TTAdapt are proposed as efficient test-time scaling methods for small VLMs.The paper reports consistent improvements over existing test-time scaling approaches with minimal overhead, suitable for resource-constrained environments.Source: §5 · p. 10
  • Experiments show both methods consistently improve performance over existing test-time scaling approaches with minimal overhead.The paper describes the methods as suitable for resource-constrained environments.Source: §5 · p. 10
  • The paper provides a systematic way to tune hyperparameters, but optimal strategies remain task- and model-dependent.Source: §5 · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility statement describes publicly available code, models, experimental details, and established evaluation benchmarks as supports for reproducing the study.

REPRODUCIBILITY STATEMENT

AUTHOR STATEMENT ON THE USE OF LARGE LANGUAGE MODELS

The authors state that large language models were used only for minor grammar and language polishing, not for scientific work, which they attribute solely to the authors.

AUTHOR STATEMENT ON THE USE OF LARGE LANGUAGE MODELS

A THEORETICAL ANALYSIS OF DIVERSITY-INDUCEMENT METHODS

The analysis models candidate quality and the model’s internal confidence jointly, then contrasts temperature sampling with greedy decoding over perturbed inputs. Under confidence-based selection, the analysis finds a larger ρσQ for the latter approach.

A THEORETICAL ANALYSIS OF DIVERSITY-INDUCEMENT METHODS

  • The framework represents candidate quality Q and internal confidence S jointly, using their correlation ρ to analyze confidence-based candidate selection.The expected maximum of N standard normal variables grows slowly as the candidate pool size N increases.Source: A THEORETICAL ANALYSIS OF… · p. 18
  • Temperature sampling yields high quality variance, but its samples often occupy low-likelihood regions where confidence aligns weakly with true quality, producing small ρ and weak scaling.Source: A THEORETICAL ANALYSIS OF… · p. 18
  • Input perturbations with greedy decoding yield lower quality variance but higher mean quality and stronger confidence–quality correlation than temperature sampling.The analysis attributes the stronger correlation to responses remaining on the likelihood manifold where the model was trained to assign high confidence.Source: A THEORETICAL ANALYSIS OF… · p. 18
  • Under confidence-based selection, ρσQ is provably larger for greedy decoding with input perturbations than for temperature sampling.The paper identifies greedy decoding with augmented inputs as a superior mechanism for generating diverse candidates in test-time scaling.Source: A THEORETICAL ANALYSIS OF… · p. 18

B THEORETICAL ANALYSIS OF TOKEN-LEVEL AGGREGATION VS. ANSWER-LEVEL AGGREGATION

The analysis establishes that token-level selection can exceed answer-level selection in expected correctness for sufficiently long responses when token selectors consistently improve base-token probabilities. Its advantage compounds across steps, making it especially relevant to longer reasoning sequences.

B THEORETICAL ANALYSIS OF TOKEN-LEVEL AGGREGATION VS. ANSWER-LEVEL AGGREGATION

  • When token selectors provide a consistent multiplicative improvement over base probabilities, token-level selection achieves higher expected correctness than answer-level selection for sufficiently long responses.The theorem states this condition and comparison directly.Source: B THEORETICAL ANALYSIS OF TOKEN-LEVEL… · pp. 18–19
  • Per-token correction prevents error propagation, compounding a (1 + δ) improvement across steps, while answer-level selection provides only a constant-factor sN improvement.The analysis contrasts multiplicative growth across steps with the constant-factor improvement from selecting among full responses.Source: B THEORETICAL ANALYSIS OF TOKEN-LEVEL… · p. 19
  • The growing advantage with sequence length makes token-level aggregation especially effective for chain-of-thought and thinking-model responses.The passage links longer responses to stepwise reasoning, where correcting errors at each token prevents their accumulation.Source: B THEORETICAL ANALYSIS OF TOKEN-LEVEL… · p. 19

C AGGREGATION METHODS

The section compares token-level aggregation strategies for test-time augmentation. Simple probability averaging performs best overall, while preserving uncertainty that discrete voting discards.

C AGGREGATION METHODS

  • The alternatives include entropy-weighted averaging, which weights predictions by confidence, majority voting, and selecting the most confident token across augmented inputs.Entropy-weighted averaging uses negative-entropy weights; the discrete approaches vote across inputs or select the highest-probability token.Source: C AGGREGATION METHODS · pp. 19–20
  • Simple averaging achieves the best overall performance among the compared token-level aggregation methods.The comparison covers averaging-based and discrete selection methods.Source: C AGGREGATION METHODS · p. 20, Table 7 · p. 20
  • Averaging-based aggregation outperforms discrete selection, and continuous probability aggregation preserves uncertainty information that discrete voting discards.Simple averaging uniformly combines probability distributions across augmented inputs.Source: C AGGREGATION METHODS · pp. 19–20

D AGGREGATION IN EARLY LAYERS

The best layer for aggregating augmented features varies by task: language-heavy tasks favor later layers, while visual reasoning benefits from earlier layers. TTAug’s parallel and sequential implementations trade memory use against runtime.

D AGGREGATION IN EARLY LAYERS

  • TTAug aggregates hidden representations at a selected transformer layer, then propagates the averaged features through the remaining layers to produce token probabilities.This tests whether intermediate semantic features or later linguistic features are better aggregation targets.Source: D AGGREGATION IN EARLY LAYERS · p. 20
  • Language-heavy benchmarks perform best with aggregation at layers 18–24, while OCRVQA and GQA favor earlier layers 6–12.ChartQA, OCRBench, and TextVQA improve monotonically toward the final output; performance on OCRVQA and GQA degrades as aggregation moves later.Source: D AGGREGATION IN EARLY LAYERS · p. 21
  • AI2D, AMBER, and COCO Captions have intermediate optima around layers 10–16, indicating that aggregation timing varies across task types.The paper relates these non-monotonic patterns to balancing semantic richness against linguistic specificity.Source: D AGGREGATION IN EARLY LAYERS · pp. 21–22
  • Although late-layer aggregation is generally stronger on average, task-specific exceptions show that final-layer features are not universally optimal.The paper suggests task-adaptive or hierarchical aggregation as possible ways to use these differences.Source: D AGGREGATION IN EARLY LAYERS · p. 22

E COMPUTATIONAL OVERHEAD OF TTAUG

  • Parallel processing keeps runtime nearly constant as augmentations increase but raises memory use; sequential processing keeps memory relatively constant but increases runtime linearly.The reported scaling comparison was measured on an NVIDIA A100 GPU.Source: E COMPUTATIONAL OVERHEAD OF TTAUG · p. 23, Figure 5 · p. 23
  • The overhead analysis uses NVIDIA A100 GPUs, though the authors describe its scaling patterns and relative trade-offs as transferable across hardware.The text attributes peak memory requirements to model architecture and batch size rather than specific hardware.Source: E COMPUTATIONAL OVERHEAD OF TTAUG · p. 23

F MULTIMODAL AUGMENTATION DECOMPOSITION

Combined text-and-image augmentation outperforms either modality alone across the reported benchmarks, with task-dependent gains and non-linear interactions. Text augmentation contributes more than image augmentation across most benchmarks.

F MULTIMODAL AUGMENTATION DECOMPOSITION

  • The ablation compares text-only, image-only, and joint augmentation, holding the other modality unchanged in each single-modality condition.The joint condition applies augmentations to both modalities.Source: F MULTIMODAL AUGMENTATION DECOMPOSITION · p. 24
  • Combined multimodal augmentation outperforms both text-only and image-only augmentation across all benchmarks, but the size of the gain varies by task.The authors report that joint gains can exceed the arithmetic sum of the single-modality gains on several benchmarks.Source: F MULTIMODAL AUGMENTATION DECOMPOSITION · p. 24
  • Text-only augmentation contributes more to performance gains than image-only augmentation across most benchmarks, especially on language-heavy tasks.The authors propose prioritizing textual diversity in resource-constrained applications while retaining image augmentation for multimodal interactions.Source: F MULTIMODAL AUGMENTATION DECOMPOSITION · pp. 24–25
  • Joint augmentation’s non-linear gains suggest that text and visual diversity can interact rather than contribute as independent additive effects.The paper describes these interactions as task-dependent and reports gains beyond the arithmetic sum on several benchmarks.Source: F MULTIMODAL AUGMENTATION DECOMPOSITION · pp. 24–25

G DETAILED RESULTS FOR DIFFERENT MODELS

This block describes model-family comparisons and implementation and evaluation details, but the supplied table and figure captions do not report numerical outcomes. It also records constraints on TTAdapt and generative image augmentation.

G DETAILED RESULTS FOR DIFFERENT MODELS

  • Tables 8–10 and Figure 7 cover results across SmolVLM2, Ovis2, InternVL2-1B, and baseline models, but their supplied captions do not state performance values.Figure 7 encodes model parameter size (B), using asinh scaling, on the x-axis and accuracy (%) on the y-axis; TTAdapt is not implemented for Ovis2 or InternVL because of library support constraints.Source: Table 8 · p. 25, Table 9 · p. 25, G DETAILED RESULTS FOR DIFFERENT MODELS · p. 25

H SMALL VISION-LANGUAGE MODELS

  • Small VLMs are described as models under 10B parameters, with examples spanning model families and sizes down to 256M parameters.The text identifies their lower computational cost as attractive for resource-constrained applications, while noting vulnerability to domain shifts and distribution mismatches.Source: H SMALL VISION-LANGUAGE MODELS · p. 26

I.1 SELF-SELECTOR

  • Self-selector asks the tested VLM to choose a candidate-response index, whereas self-synthesizer asks it to combine candidate responses into one answer.Structured generation is used to ensure valid index responses for self-selector.Source: §I.1 · pp. 26–27, §I.2 · p. 27

I.3 SELF-PARAPHRASING

  • Self-paraphrasing splits prompts into sentences, paraphrases each independently, then combines the paraphrase sets; classical image augmentation randomly applies three transformations.Generative image augmentation uses FLUX.1-dev to make visually distinct variants, but excludes text-containing images to prevent OCR corruption and requires external diffusion models and substantial computation.Source: §I.3 · p. 27, §I.4 · p. 28, §I.5 · p. 29

I.6 AGGREGATION WEIGHTS OPTIMIZATION

  • The learnable aggregation variant optimizes token-wise augmentation weights to minimize entropy, using detached forward-pass probabilities to keep the computational graph compact.Weights are initialized uniformly and reinitialized for each question; the supplied configuration uses 20 optimization micro-steps per token.Source: §I.6 · pp. 29–30

I.7 MODEL PARAMETER ADAPTATION

  • TTAdapt iteratively fine-tunes model parameters using TTAug-consensus pseudolabels, then resets weights between questions; TTAug intercepts generation to aggregate logits while preserving KV caching.Evaluation covers nine benchmarks and uses task-specific metrics, including exact matching, VQA scoring, relaxed string matching, substring containment, choice-label matching, and ROUGE-L; preprocessing normalizes text consistently.Source: §I.7 · p. 30, §I.8 · p. 30, J EVALUATION METRICS DETAILS · p. 31

K STANDARD ERROR VALUES

The appendix specifies how reported standard errors are estimated and summarizes comparisons among test-time scaling strategies.

K STANDARD ERROR VALUES

  • Standard errors are estimated from the empirical variance of per-sample scores, and the sample mean is the reported average accuracy.The standard error estimate follows the Central Limit Theorem.Source: K STANDARD ERROR VALUES · p. 32
  • TTAug outperforms all compared test-time scaling methods across accuracy and efficiency metrics.The comparison is reported in Table 11.Source: Table 11 · p. 33
  • Input Perturbation outperforms Temperature Sampling among diversity-inducement methods.The comparison is against the Baseline, as reported in Table 12.Source: Table 12 · p. 33
  • Token-level aggregation outperforms all other compared answer-level and token-level aggregation approaches.The comparison is reported in Table 13.Source: Table 13 · p. 33
  • Self-Paraphrasing 2 and Classical Augmentations 3 consistently perform best among text augmentation strategies.The comparison is reported in Table 14.Source: Table 14 · p. 34
  • Model parameter adaptation 2 yields the best performance among the compared test-time adaptation strategies.The comparison is reported in Table 16.Source: Table 16 · p. 34

L QUALITATIVE RESULTS

Qualitative examples illustrate how TTAug and TTAdapt affect outputs across samples, including cases where they correct baseline errors or improve reported caption metrics, as well as cases where TTAug does not improve.

L QUALITATIVE RESULTS

  • The displayed cases apply 16 classical text and high-strength image augmentations per sample and correspond to samples underlying Sec. 4.7's quantitative results.Source: L QUALITATIVE RESULTS · p. 35
  • 100.0% accuracy for TTAug and TTAdapt on the window question contrasts with 0.0% for the baseline; both methods answer “Desk” while the baseline answers “Blinds.”The question asks what is in front of the window.Source: L QUALITATIVE RESULTS · p. 38
  • 100.0% accuracy is reported for all three outputs on the cricket-population question, with each answering A.Option A is “lizards would decrease.”Source: L QUALITATIVE RESULTS · p. 40
  • 100.0% accuracy is shown with answer B in each of two displayed outputs.The supplied passages do not identify which of the three labeled methods corresponds to each of these two outputs.Source: L QUALITATIVE RESULTS · p. 41
  • 62.50 BLEU-1, 40.82 BLEU-2, 32.93 BLEU-3, 27.23 BLEU-4, 53.20 ROUGE-L, and 60.98 CIDEr are reported for TTAug’s caption.The caption describes a white bird with a long beak and black legs standing on a green bench.Source: L QUALITATIVE RESULTS · p. 43
  • 52.38 BLEU-1, 36.19 BLEU-2, 27.45 BLEU-3, 21.89 BLEU-4, 45.02 ROUGE-L, and 12.62 CIDEr are reported for TTAdapt’s caption.The caption describes a white bird with a long curved beak standing on a green bench.Source: L QUALITATIVE RESULTS · p. 43

Related papers

Loading 2510.03574v2…