PaperlayerView exact revision on arXiv

Source-linked AI summary

LEAML: Label-Efficient Adaptation to Out-of-Distribution Visual Tasks for Multimodal Large Language Models

Ci-Siang Lin, Min-Hung Chen, Yu-Yang Sheng, Yu-Chiang Frank Wang

arXiv:2510.03232v1cs.CV

TL;DR

MLLMs can struggle with specialized out-of-distribution VQA, where labeled examples are limited and costly. LEAML generates domain-relevant pseudo QA pairs from unlabeled images and selectively updates QA-relevant neurons. It improves gastrointestinal endoscopy VQA over standard fine-tuning, while generated QA quality remains constrained by the small labeled set.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have achieved strong performance on general visual benchmarks but struggle with out-of-distribution (OOD) tasks in specialized domains such as medical imaging, where labeled data is limited and expensive. We introduce LEAML, a label-efficient adaptation framework that leverages both scarce labeled VQA samples and abundant unlabeled images. Our approach generates domain-relevant pseudo question-answer pairs for unlabeled data using a QA generator regularized by caption distillation. Importantly, we selectively update only those neurons most relevant to question-answering, enabling the QA Generator to efficiently acquire domain-specific knowledge during distillation. Experiments on gastrointestinal endoscopy and sports VQA demonstrate that LEAML consistently outperforms standard fine-tuning under minimal supervision, highlighting the effectiveness of our proposed LEAML framework.

Introduction

MLLMs perform well on general visual benchmarks but can be unreliable on specialized out-of-distribution tasks, where VQA labels are costly and full fine-tuning can overfit. LEAML uses labeled QA and unlabeled images to generate domain-relevant training pairs and adapt the target model.

Introduction

  • Specialized-domain MLLMs can produce unreliable outputs, while expert QA annotation is costly, random question-image pairings are confusing, and full fine-tuning can overfit scarce data.Source: Introduction · p. 1
  • LEAML trains a QA Generator on scarce labeled pairs, regularizes it with caption distillation, and uses it to create domain-relevant pseudo QA pairs for target-model fine-tuning.Selective Neuron Distillation updates knowledge-relevant neurons in the generator.Source: Introduction · pp. 1–2
  • LEAML significantly improves gastrointestinal endoscopy VQA performance over standard fine-tuning approaches.The framework uses scarce labeled VQA data and unlabeled images, with QA-related neuron updates as part of its approach.Source: Introduction · pp. 1–2

Related Works

Prior work addresses OOD detection and fixed-label classification, while domain-specific VQA raises distinct challenges in producing grounded questions and answers. Related work also motivates semi-supervised learning and targeted interventions on knowledge-bearing neurons.

Out-of-Distribution Data Learning

  • OOD research has focused on detection or fixed-label classification, while retrieval and caption-based QA approaches retain assumptions about foundational domain knowledge or caption granularity.Caption-based QA quality depends on both the LLM's domain knowledge and the detail in source captions.Source: Out-of-Distribution Data Learning · p. 2

Semi-Supervised Learning

  • Pseudo-labeling works for discrete labels, but VQA also requires a grounded answer and a relevant, context-aware question aligned with each image.Randomly assigning questions to images can create incoherent or misleading training data.Source: Semi-Supervised Learning · p. 2

Neuron-Level Knowledge Attribution in DNNs

  • Prior findings associate Transformer FFN layers and specialized neurons with stored knowledge, motivating targeted manipulation rather than uniformly changing all parameters.Attribution methods also guide fine-tuning by selecting components, commonly using gradient magnitudes as an importance signal.Source: Neuron-Level Knowledge Attribution in… · p. 2

Method

LEAML adapts a pretrained MLLM to specialized-domain VQA with few labeled examples and abundant unlabeled images. It generates pseudo QA pairs for unlabeled images, using caption distillation and selective updates to focus the QA Generator on QA-related knowledge.

Problem Formulation and Framework Overview

  • LEAML adapts a pretrained MLLM in two stages: generate pseudo QA pairs for unlabeled images, then fine-tune with those pairs and scarce labeled data.The setting includes a few labeled image-or-video, question, and answer samples alongside abundant unlabeled visual inputs.Source: Problem Formulation and Framework… · p. 3

Selective Neuron Distillation for QA Generator

Pseudo QA Generation and OOD VQA Finetuning

  • The QA Generator learns from labeled examples by generating concatenated questions and answers from images, then produces pseudo QA pairs for unlabeled images.Those generated pairs augment the labeled data for fine-tuning the final VQA model; their quality remains tied to the small labeled set.Source: Pseudo QA Generation and OOD VQA… · p. 4

Selective Neuron Distillation for QA Generator

  • A large pretrained MLLM supplies captions for unlabeled images, and the QA Generator combines caption-distillation and labeled-data QA objectives.The captions are used to provide domain-related knowledge during generator training.Source: Selective Neuron Distillation for QA… · pp. 4–5

LG = LQA + LC

The generator combines labeled-data QA training with caption distillation while restricting updates to parameters most relevant to QA generation.

Selective Neuron Distillation for QA Generator

  • QA-loss gradient magnitude scores identify important parameters, and only the top-K parameters in each neuron are updated while the rest remain frozen.This focuses caption-distilled knowledge on QA-related capacity rather than generic captioning or unrelated parameters.Source: LG = LQA + LC · p. 5

Experiments

Experiments use gastrointestinal endoscopy and sports VQA with only 1% of training QA pairs labeled. LEAML improves accuracy over limited-label fine-tuning baselines, and ablations favor combining caption distillation with selective neuron updates.

Kvasir-VQA

  • Kvasir-VQA training and testing splits contain 18,499 and 18,075 QA pairs; only 1% of training pairs are labeled, and answers use multiple-choice options.Kvasir-VQA contains expert-annotated QA pairs for endoscopic images.Source: Kvasir-VQA · p. 5

SPORTU

  • SPORTU splits contain 5,525 training and 5,478 testing QA pairs, with only 1% of training pairs labeled.Its questions span seven sports and three difficulty levels, including hard questions requiring rule reasoning and foul detection.Source: SPORTU · p. 5

Quantitative and Qualitative Experiments

Implementation Details

  • The QA Generator and VQA model share an MLLM backbone, while captioning distillation uses Qwen2.5-VL-72B-Instruct; neuron selection sets K = 1000 for Kvasir-VQA and K = 3000 for SPORTU.Source: Implementation Details · p. 6

Ablation Studies

  • Caption distillation alone improves Kvasir-VQA average accuracy by 0.6% over baseline; selective updates yield a 3.4% total gain, with a similar SPORTU trend.The reported gains indicate that substantial improvement occurs when both components are used together.Source: Table 3 · p. 7, Table 4 · p. 7, Ablation Studies · p. 7

Quantitative and Qualitative Experiments

Conclusion

LEAML adapts MLLMs to OOD domains using labeled VQA data and unlabeled images. Its experiments report improvements over conventional fine-tuning on domain-specific visual tasks.

  • LEAML combines scarce labeled data with abundant unlabeled images to adapt MLLMs to domain-specific OOD visual tasks.Source: Conclusion · p. 7
  • Pseudo QA Generation creates domain-relevant question-answer pairs from unlabeled visual data to expand training.Source: Conclusion · p. 7
  • Selective Neuron Distillation updates knowledge-relevant neurons in the QA Generator while it acquires domain knowledge from large MLLMs.Source: Conclusion · p. 7
  • Across gastrointestinal endoscopy and sports VQA, experiments report improvements over conventional fine-tuning, supported by ablations and quantitative and qualitative analyses.Source: Conclusion · p. 7

Additional Visualization

The additional visualizations compare LEAML with baselines on medical and sports VQA and show qualitative generated pseudo-QA and captions across Kvasir-VQA and SPORTU.

Comparison with Baseline in VQA

  • Qualitative comparisons report that LEAML produces more accurate, domain-specific answers than baseline approaches on medical and sports VQA.The reported gains include better understanding of specialized terminology and visual patterns.Source: Comparison with Baseline in VQA · p. 10

Captions from Large MLLM

Loading 2510.03232v1…