PaperlayerView exact revision on arXiv

Source-linked AI summary

When Big Models Train Small Ones: Label-Free Model Parity Alignment for Efficient Visual Question Answering using Small VLMs

Abhirama Subramanyam Penamakuri, Navlika Singh, Piyush Arora, Anand Mishra

arXiv:2509.16633v1cs.CVcs.AIcs.CL

TL;DR

L-VLMs are costly for inference-heavy VQA, while more efficient S-VLMs lag in performance. MPA uses unlabeled images and parity-targeted training; it improves S-VLM performance across four VQA benchmarks.

  • Problem

    L-VLMs’ computational requirements and latency constrain their practicality for inference-heavy VQA, while efficient S-VLMs fall short in performance.

    Source: §1 · p. 1

  • Method

    MPA uses unlabeled images and L-VLM guidance to target knowledge gaps, then trains the S-VLM on parity samples.

    Source: §1 · p. 2, §3 · p. 3, §3.3 · p. 5

  • Results

    15.2% maximum and 3.4% average absolute gain; MPA improves all S-VLMs on all four datasets.

    Source: §4.1 · p. 6

  • Takeaways & Limitations

    The evaluation spans TextVQA, ST-VQA, ChartQA, and OKVQA, covering varied VQA benchmark settings.

    Source: §5 · p. 9

    MPA requires access to an L-VLM; proprietary APIs may constrain reproducibility and transparency, and experiments focus primarily on English-language VQA.

    Source: Limitations · pp. 9–10

Abstract

from arXiv · show

Large Vision-Language Models (L-VLMs) have demonstrated remarkable performance in various vision and language tasks, including visual question answering (VQA). However, their high computational cost makes them impractical for resource-constrained settings and inference-heavy applications. In contrast, Small Vision-Language Models (S-VLMs) offer efficiency but suffer from a significant performance gap compared to their larger counterparts. In this work, we introduce the Model Parity Aligner (MPA), a novel framework designed to systematically improve S-VLMs by leveraging unlabeled images and effective knowledge transfer from L-VLMs. Instead of traditional knowledge distillation methods that rely on labeled training data, MPA employs a strategic parity-based approach that precisely identifies the knowledge disparities between S-VLMs and L-VLMs, and optimizes training by targeting only these disparities. We conduct extensive experiments on four diverse VQA benchmarks, namely TextVQA, ST-VQA, ChartQA, and OKVQA, each of which requires specialized reasoning capabilities such as text recognition, chart interpretation, and commonsense and factual understanding. Our results demonstrate that MPA consistently enhances the performance of S-VLMs on all benchmarks, reducing the performance gap while maintaining computational efficiency. We make our code publicly available.

1 Introduction

L-VLMs offer strong VQA performance but are costly for inference, while efficient S-VLMs lag behind. MPA uses unlabeled images to transfer knowledge by targeting gaps between large and small models.

1 Introduction

  • L-VLMs’ high computational requirements and latency limit their practicality for resource-constrained, inference-heavy VQA, while S-VLMs are more efficient but perform worse.This motivates the question of whether relevant knowledge transfer can improve S-VLMs.Source: §1 · p. 1
  • Traditional knowledge distillation typically relies on labeled data, adapters still require large-model inference, and naïve pseudo-labeling can propagate noise.The introduction identifies systematic pseudo-label-based transfer from large to small VLMs as underexplored.Source: §1 · pp. 1–2
  • MPA transfers knowledge from L-VLMs using unlabeled images, identifying and addressing S-VLM knowledge gaps with quality-assessed pseudo-labels.The introduction describes this as a way to improve performance while maintaining computational efficiency.Source: §1 · p. 2
  • Experiments report consistent S-VLM improvements across four diverse VQA benchmarks, including TextVQA, ST-VQA, ChartQA, and OKVQA.These benchmarks test visual-text understanding, chart interpretation, and world-knowledge integration.Source: §1 · p. 2
  • The authors also report benefits beyond VQA, including improved text recognition and text-aware captioning.Source: §1 · p. 2

2 Related Work

Related work frames VLM development around the trade-off between model capacity and computational demands. MPA is distinguished by selecting knowledge-gap examples for transfer and quality-checking pseudo-labels.

Small and Large VLMs

  • Growing parameter counts in large VLMs constrain computational efficiency, accessibility, and deployment costs, motivating smaller models that seek competitive performance with lower demands.The paper uses a parameter-based taxonomy that classifies VLMs with ≤5B parameters as small and those above that threshold as large.Source: §2 · p. 2

Knowledge Distillation

  • Unlike standard knowledge distillation over labeled data, MPA supervises only samples where the student and teacher differ, using input-output access that can include closed-source L-VLMs.Source: §2 · p. 3

Data Augmentation for VQA

  • Unlike prior VQA augmentation approaches, MPA quality-checks L-VLM pseudo-labels to discard noise and target improvements with selected annotations.Source: §2 · p. 3

3 Model Parity Aligner (MPA)

MPA turns unlabeled images into task-specific pseudo-annotations, selects examples where the L-VLM is right and the S-VLM is wrong, then fine-tunes the S-VLM on that parity subset.

3 Model Parity Aligner (MPA)

  • MPA takes an L-VLM, an S-VLM, unlabeled images, and a task, and returns an enhanced S-VLM whose model size remains much smaller than the L-VLM’s.Its workflow comprises the Pseudo Annotator, Parity Identifier, and Parity Leveler.Source: §3 · p. 3

3.1 Pseudo Annotator (PA)

  • The Pseudo Annotator prompts an L-VLM to generate task-specific question-answer pairs for unlabeled images; prompts can target chart reasoning or scene-text understanding.Automated annotations may be noisy or hallucinated, so they require quality validation.Source: §3.1 · p. 4

3.2 Parity Identifier (PI)

  • The Parity Identifier selects samples where the L-VLM matches the pseudo-annotation and the S-VLM does not, filtering noisy annotations and focusing on a subset much smaller than the full set.The paper presents this subset as samples focused on the S-VLM’s specific knowledge deficiencies.Source: §3.2 · p. 5

3.3 Parity Leveler (PL)

  • The Parity Leveler fine-tunes the S-VLM on PI-selected parity samples using minibatch generation loss and stochastic gradient descent, while L-VLM parameters remain frozen.Source: §3.3 · p. 5

4 Experiments and Results

MPA is evaluated on four VQA benchmarks with five small and three large VLMs, and improves S-VLM performance across the tested combinations. The analyses cover task and model differences, comparisons with supervised baselines, annotation quality, and transfer beyond standard VQA.

4.1 Results and Discussion

  • MPA improves all S-VLMs on all four benchmarks, with a maximum absolute gain of 15.2% and an average gain of 3.4%.The evaluation covers TextVQA, ST-VQA, ChartQA, and OKVQA.Source: §4.1 · p. 6, Table 1 · p. 6
  • TinyLLaVA-2B reaches 27.2% accuracy on ChartQA with InternVL2-8B guidance, a +15.2% absolute gain; two S-VLMs also gain +4.7% on ST-VQA.The ST-VQA gains are for Qwen2VL-2B guided by Qwen2VL-7B and InternVL2-4B guided by InternVL2-8B.Source: §4.1 · p. 6, Table 1 · p. 6
  • Gains vary by task: the reported examples include +6.4% on TextVQA and +5.1% on ST-VQA, while OKVQA gains are modest.The authors attribute the modest OKVQA gains to reliance on external knowledge, which MPA cannot fully address without RAG or knowledge-rich fine-tuning.Source: §4.1 · p. 7
  • MPA improves all tested model scales; TinyLLaVA-2B has the highest average gain (+6.8), while InternVL2-4B gains +2.1, and Qwen2VL-7B is the strongest reported guide.Across five S-VLMs and four datasets, Qwen2VL-7B averages +3.5 points versus +3.2 for InternVL2-8B; the authors cite pretraining gaps and diminishing returns as possible explanations for scale differences.Source: §4.1 · p. 7

4.1.1 Ablations and Analysis

  • Beyond VQA, MPA raises visual text recognition accuracy by 4.5% and improves text-aware captioning metrics; on binary PathVQA, it yields a +2.4% gain.The PathVQA evaluation focuses on yes/no questions because open-ended items contain specialized medical terminology.Source: §4.1.1 · pp. 7–8, Table 3 · p. 7, Table 7 · p. 8
  • MPA outperforms LoRA-based SFT and full SFT using 100 labeled samples, and GPT-4o guidance improves S-VLMs without access to its logits or weights.With GPT-4o guidance, MPA-aligned Qwen2VL-2B scores 75.4% on TextVQA versus 74.7% for Qwen2VL-7B.Source: Table 2 · p. 6, §4.1.1 · pp. 7–8, Table 5 · p. 8
  • The PI-selected subset achieves the highest reported accuracy across TextVQA (75.1%), ST-VQA (67.2%), and ChartQA (67.6%), despite using fewer samples.A user study also reports higher post-PI annotation quality across all evaluated metrics, with more annotations identified as human-annotated.Source: Table 4 · p. 7, §4.1.1 · p. 8, Table 6 · p. 8
  • Manual inspection identifies student errors in OCR grounding, chart-entity alignment and trend interpretation, and internal-knowledge grounding or visual guesswork.The analysis samples 100 DPI examples per task and excludes verbose or noisy/task-irrelevant cases from its knowledge-gap categories.Source: §4.1.1 · p. 9

5 Conclusion and Future Work

MPA uses unlabeled images and knowledge transfer from L-VLMs to improve S-VLMs through pseudo-labeling with quality assessment. It improves S-VLM performance across four VQA benchmarks, while a gap with L-VLMs remains.

5 Conclusion and Future Work

  • MPA uses unlabeled images and L-VLM knowledge transfer, with quality-assessed pseudo-labeling rather than traditional distillation on labeled data.The conclusion describes high-confidence supervision as a way to avoid error propagation.Source: §5 · p. 9
  • MPA improves S-VLMs across TextVQA, ST-VQA, ChartQA, and OKVQA, making them more viable for real-world applications with limited resources.The framework uses unlabeled images and pseudo-labeling with quality assessment.Source: §5 · p. 9
  • A performance gap between S-VLMs and L-VLMs remains, motivating further work on alignment and extensions beyond VQA.Proposed directions include iterative pseudo-label refinement, diverse unlabeled data, and integrating L-VLM multi-step reasoning.Source: §5 · p. 9

Limitations

The framework depends on an L-VLM to generate and validate pseudo-annotations, which may constrain its use in stricter resource-limited settings. Evaluation is focused primarily on English-language datasets and VQA-related tasks.

Limitations

  • MPA depends on access to an L-VLM for generating and validating pseudo-annotations, which may limit applicability in stricter resource-constrained settings.When proprietary models are accessed through commercial APIs, limited insight into model behavior and changing responses may compromise reproducibility and transparency.Source: Limitations · p. 9
  • Experiments focus primarily on English-language datasets and VQA-related tasks; generalization to multilingual or more complex reasoning tasks remains open.Source: Limitations · p. 10

Ethical Considerations and Broader Impact

The paper notes potential bias in both open-source datasets and L-VLM outputs, and says the Parity Identifier cannot eliminate inherited bias. It also describes its human evaluation safeguards and the potential for MPA to support low-resource applications.

Ethical Considerations and Broader Impact

  • Open-source datasets may contain social or cultural biases, and L-VLM outputs may also be hallucinated or biased.The Parity Identifier filters low-quality or incorrect annotations but cannot entirely eliminate inherited biases.Source: Ethical Considerations and Broader… · p. 10
  • Three annotators assessed pseudo-annotation quality and were compensated fairly according to local wage norms.They were not exposed to harmful, offensive, or sensitive content, and no personally identifiable information was collected.Source: Ethical Considerations and Broader… · p. 10
  • The authors state that training S-VLMs with unlabeled data reduces reliance on expensive human annotations and may broaden access in low-resource settings.Potential application areas named include healthcare, agriculture, and accessibility.Source: Ethical Considerations and Broader… · p. 10

A Additional Analysis

Additional analyses report gains from Parity Identifier filtering, transfer to OCR and text-aware captioning, and estimated computational and API costs for the pipeline.

Additional comparisons: utility of PI filtering

  • MPA outperforms MPA without PI across tasks and S-VLMs while using fewer training samples, including about 2K versus 21K for TextVQA.The authors interpret this as evidence that PI filtering isolates knowledge-gap samples for more efficient, targeted supervision.Source: A Additional Analysis · p. 14

Expanded comparison on OCR and text-aware captioning tasks

  • On OCR and text-aware image captioning, MPA-trained models improve over zero-shot baselines and surpass models trained on human-labeled TextVQA data.The analysis uses ICDAR 2015 visual text recognition and TextCaps captioning evaluations for TinyLLaVA-2B aligned with Qwen2VL-7B.Source: A Additional Analysis · p. 14

Computational and API cost of PA and PI

  • For TextVQA, generating about 21K pseudo-annotations with local Qwen2VL-7B takes 4–6 hours on three A6000 GPUs, and PI takes another 2–3 hours.The estimated PA + PI API cost using GPT-4o is about $11 per S-VLM–task pair.Source: A Additional Analysis · p. 14

B Dataset Details

The section describes the size and splits of four VQA datasets, including TextVQA’s validation-set reporting because its test set is private.

B Dataset Details

  • TextVQA has 28K images and 45K question-answer pairs; because its test set is private, results are reported on the validation set.Its splits include 21K training images, 3K validation images, and a private test set.Source: B Dataset Details · p. 14
  • ST-VQA contains 23K images and 31K questions, while ChartQA contains 21.6K charts and 32.3K question-answer pairs.ST-VQA has 16K training images and 2.8K test images; ChartQA has 19K training charts, 1K validation charts, and 1.6K test charts.Source: B Dataset Details · p. 14
  • OKVQA consists of 14K images and 14K questions, with 9K questions for training and 5K for testing.Source: B Dataset Details · p. 14

C Implementation Details

Implementation trains the S-VLM using PI-module samples and uses dataset-specific prompts to generate pseudo-annotations. The section also reports compute requirements and an OCR comparison.

C Implementation Details

  • The parity leveler trains the entire S-VLM for one epoch on samples from the PI module across all benchmark datasets.Experiments use the original or Hugging Face model implementations, depending on reproducibility; hyperparameters are summarized in Table 11.Source: C Implementation Details · p. 14, Table 11 · p. 15
  • Each L-VLM and S-VLM combination takes approximately 5–12 GPU hours per dataset on a machine with three 48 GB Nvidia A6000 GPUs.Source: C Implementation Details · p. 14

D Prompts used

  • Prompts generate pseudo-annotations for all four datasets, with chart prompts covering lookup, comparison, trends, and inference questions.The chart prompt requests a question and answer derived from chart data.Source: D Prompts used · pp. 14–15
  • Without ground-truth labels, MPA outperforms both the zero-shot baseline and human-labeled models on OCR and text-aware captioning.Source: Table 10 · p. 15

E Qualitative Results

The qualitative examples show MPA-aligned S-VLMs correcting some zero-shot errors across VQA tasks. The PI module also discards examples that show no knowledge gap or are noisy.

E Qualitative Results

  • MPA alignment corrects some zero-shot S-VLM errors and improves performance in visual text reasoning, plot interpretation, and knowledge-based question answering.The paper shows additional examples from TextVQA, ST-VQA, ChartQA, and OKVQA.Source: E Qualitative Results · pp. 15–16
  • The PI module discards pseudo-annotations that do not represent an S-VLM–L-VLM disparity and annotations identified as noisy.Figure 4 shows examples of both categories.Source: E Qualitative Results · p. 16, Figure 4 · p. 16
  • Figure 5 compares zero-shot and MPA-aligned Qwen2VL-2B outputs, using Qwen2VL-7B as the L-VLM; green and red mark correct and incorrect answers.Source: Figure 5 · p. 17
  • Figures 6–9 provide further comparisons between baseline and MPA-aligned S-VLM outputs for TextVQA, ST-VQA, ChartQA, and OKVQA.The captions describe these examples as showing the aligned model’s efficiency, but do not provide numerical results.Source: Figure 6 · p. 17, Figure 7 · p. 17, Figure 8 · p. 18

Related papers

Loading 2509.16633v1…