PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Evolving Visual Questioner

Yijun Liang, Hengguang Zhou, Ming Li, Lichen Li, Cho-Jui Hsieh, Tianyi Zhou

arXiv:2606.13929v1cs.CVcs.LG

TL;DR

Visual question generation by VLMs remains underexplored because they are mainly trained as passive answerers. This paper introduces a self-evolving framework that generates and filters its own visual questions, increasing average QG score by ∼82% after two rounds while largely preserving QA performance.

  • Problem

    VLMs are primarily trained as passive answerers, leaving their ability to ask informative, diverse, and visually grounded questions comparatively underexplored.

  • Method

    A VLM iteratively proposes, rewrites, filters, and trains on diverse, difficult, visually grounded questions generated from unlabeled images.

  • Results

    After two self-evolving rounds, average QG score increases by ∼82% relative to the initial base model while largely preserving QA performance.

  • Takeaways & Limitations

    Self-supervision can substantially improve autonomous visual question generation while maintaining competitive answerer performance.

  • Takeaways & Limitations

    The proposal–rewriting–filtering–training loop adds computation compared with directly training on existing QA data, especially when applied iteratively.

Abstract

from arXiv · show

Vision-language models (VLMs) are typically trained as passive answerers, while their ability to actively ask diverse, non-trivial, visual-centric and grounded questions remains underexplored. Existing visual questioners' performance is bottlenecked by the availability of high-quality training data or the cost of curating them. We show that a VLM can continuously improve itself as a visual questioner without any external supervision. We propose a self-evolving framework that uses a VLM itself as both a proposer and a filter to produce harder, more informative, and visual-centric questions, while maintaining their exploration diversity to avoid training collapse. These questions are then used to train the VLM in both questioner and answerer modes. To evaluate the questioner, we introduce an agentic protocol that assesses questions along perception, reasoning, and diversity dimensions. Experiments across various backbone VLMs show that our method substantially enhances the quality and substantially expands the difficulty boundary of autonomous question generation. Under the same budget, our self-supervision is more effective than training on the static source data. Moreover, the self-evolving questioner remains a competitive or even better answerer.

1 Introduction

The paper frames visual questioning as an underexplored capability because VLMs are mainly trained as passive answerers and existing question-generation methods depend on static supervision. It proposes a fully autonomous self-evolving framework that uses a VLM to generate, improve, and filter questions while evaluating perception, reasoning, and diversity.

  • Motivation: Most VLMs treat questions as fixed inputs and focus on producing better answers, leaving active visual questioning underexplored.Questioning requires inspecting images, identifying informative visual evidence, and formulating perception- and reasoning-oriented queries.
  • Problem: Existing visual question generation methods are bottlenecked by static data distributions and tend to imitate patterns from human, curated, or external-model supervision.This limits the diversity, grounding quality, and reasoning complexity of generated questions.
  • Method: Multi-round generation, rewriting, and filtering promote diverse, difficult, and visually grounded questions while preventing degeneration during recursive self-training.The framework explicitly sustains exploration diversity and progressively expands reasoning difficulty throughout training.
  • Method: The proposed framework enables a VLM to improve as a visual questioner without human annotations, external teacher models, or auxiliary reward models.It iteratively constructs stronger supervision from unlabeled images using the current model itself.
  • Evaluation and Results: An agentic evaluation protocol measures generated questions along perception, reasoning, and diversity dimensions beyond conventional QA accuracy.Experiments show improved visual-centric question quality while largely preserving QA performance, with stable or improved accuracy on several benchmarks.

2 From Answerer to Questioner

The framework turns a VLM from an answerer into a self-evolving visual questioner by generating, refining, filtering, and training on its own question-answer data. It evaluates questions for visual evidence, reasoning difficulty, and set-level diversity.

  • Self-evolving framework: The framework proposes candidate questions, rewrites them into harder visual-centric questions, and filters them for answerability, visual grounding, and perception/reasoning difficulty.Retained question-answer pairs support dual-format training: QA supervision anchors answering, while QG supervision teaches question-and-answer generation.
  • Question generation: Visual question generation conditions question production on the image and optional controls such as question type, answer style, or reasoning intent.These controls encourage broader coverage of the per-image question space; without control, questions are generated directly from the image.
  • Evaluation protocol: The protocol evaluates individual questions through perception difficulty, covering visual search and visual evidence coverage, and reasoning difficulty, covering contextual and spatial reasoning.These dimensions assess the visual evidence required and how that evidence must be interpreted.
  • Evaluation protocol: Question-set evaluation measures questioning diversity, capturing redundancy among questions generated for the same image.This prevents valid but near-duplicate questions from providing little additional supervision.
  • Evaluation protocol: A GPT-based judge scores individual questions conditioned on the image, while sentence embeddings measure semantic distances among questions for set-level diversity.The protocol therefore emphasizes visual evidence and reasoning requirements rather than surface wording.

3 Self-Evolving Visual Questioner

The framework self-evolves visual question generation through proposer-driven candidate construction, harder visual-centric rewriting, model-based filtering, and dual-format training. Iteratively, it expands question difficulty and diversity while preserving answering ability through QA supervision.

  • Framework overview: The framework has two stages: self-supervision data construction and dual-format model training, using the VLM in multiple roles to improve questioning while preserving answering.Given unlabeled images, the model proposes candidates, rewrites them, filters them, and trains on retained supervision.
  • Question proposal: The evolving model proposes answerable, visually grounded questions across recognition, comparison, spatial, scene-understanding, and grounded-reasoning intents.Multiple intents broaden image coverage and reduce shallow or repetitive templates.
  • Question rewriting: The initial checkpoint rewrites proposals toward greater visual inspection, evidence grounding, contextual reasoning, or spatial reasoning, adding exploration without external supervision.Separating proposal from rewriting avoids coupling both steps to the same adapted generation distribution.
  • Candidate filtering: Filtering retains visually answerable, image-grounded candidates that improve perception or reasoning difficulty, discarding ambiguous, weakly grounded, unanswerable, or non-improving generations.The retained candidates form the training dataset for subsequent adaptation.
  • Dual-format training: Dual-format supervision trains question-and-answer generation from images while answering questions from image-question inputs, improving question generation and anchoring answering behavior.Across iterative rounds, rewriting, filtering, and dual-format training form a self-improvement loop that maintains competitive downstream QA performance.

4 Experiments

Experiments across three VLM backbones show that self-evolution improves all evaluated question-generation dimensions while preserving downstream answering ability. Iterative refinement, dual-format supervision, rewriting plus filtering, and improved questions each contribute to stronger results under small-data budgets.

  • Experimental setup: Experiments evaluate Qwen2.5-VL-3B-Instruct, Qwen2.5-VL-7B-Instruct, and Qwen3VL-4B-Instruct at base, first-round, and second-round stages using 10K QA+QG examples.Question generation uses five questions per image and five normalized dimensions; downstream answering is evaluated on four benchmarks.
  • Main results: Self-evolution consistently improves all five QG dimensions across backbones, including the strong Qwen3VL-4B baseline, without degrading downstream QA performance.Under dual-format training, 10K self-supervision examples improve question generation while preserving answering behavior.
  • Iterative self-improvement: Second-round adaptation further improves all five QG dimensions beyond first-round adaptation, showing continued gains from iterative self-improvement.Later-round models generate stronger proposals that are refined again rather than receiving only a one-time benefit.
  • Ablations: QA+QG offers the best trade-off among supervision formats: it improves QG average over QA-only across all backbones while achieving the best or comparable QA performance.QG-only generally improves QG over QA-only but can reduce downstream answering ability.
  • Ablations: Adding filtering to rewriting raises visual search from 0.353 to 0.433, evidence coverage from 0.520 to 0.547, contextual reasoning from 0.507 to 0.553, spatial reasoning from 0.293 to 0.367, and diversity from 0.320 to 0.350.The full pipeline achieves the strongest QG performance across all dimensions, indicating that filtering selects candidates with stronger grounding and visual reasoning.
  • Downstream QA: 61.90% to 63.32%: training with questions from the improved QG model raises average QA accuracy, including CVBench-3D from 69.25% to 75.58%.Improved-question training also performs better on CVBench-2D, VStar, and RWQA.

5 Related Work

Prior visual question generation relies mainly on externally supervised or controllable data and is often evaluated with reference-based metrics. Recent self-improvement methods reduce supervision dependence, but existing multimodal judges do not primarily assess fine-grained image-grounded questioning.

  • Visual Question Generation: Existing VQG methods commonly use external supervision from VQA-style data, LLM-generated questions, or controllable data with predefined constraints.Constraints can specify answers, regions, knowledge, or difficulty levels.
  • VQG Evaluation: VQG evaluation often compares generated questions with references using BLEU, METEOR, and CIDEr.These metrics measure similarity to given ground-truth questions.
  • VQG Evaluation: Recent LLM- or MLLM-based judges evaluate multimodal outputs or generated questions, but mainly assess general response or text-domain quality rather than fine-grained image grounding.This leaves fine-grained image-grounded questioning capability insufficiently evaluated.
  • Self-Improvement: Post-training traditionally depends on instruction data from humans, curated datasets, or stronger models, while recent self-improvement methods generate, filter, critique, or judge their own training signals.Self-improvement reduces dependence on externally constructed supervision, whose quality is costly and difficult to scale.

6 Conclusion

The paper presents a self-evolving visual questioner that improves VLM question generation without human annotations or external teacher models. It builds visual-centric, diverse, reasoning-oriented supervision from unlabeled images and evaluates question generation across perception, reasoning, and diversity.

  • Conclusion: The framework improves VLM question generation without human annotations or external teacher models.It uses a self-evolving training approach rather than externally curated supervision.
  • Conclusion: Question proposal, rewriting, filtering, and dual-format QA/QG training construct visual-centric, diverse, reasoning-oriented supervision from unlabeled images.The framework combines these stages to generate training supervision internally.
  • Conclusion: The evaluation protocol assesses question generation across perception, reasoning, and diversity dimensions.Experiments show substantial question-generation improvements while largely preserving question-answering performance.

7 Limitations

The framework’s filtering criteria do not yet fully capture visual grounding, ambiguity, or instructional value, and its iterative evolution loop adds computational overhead. Future work could refine filtering and reduce this cost through batching, lightweight filters, or selective evolution strategies.

  • Filtering limitations: Current filtering emphasizes answerability and perception/reasoning difficulty rather than fine-grained visual grounding, ambiguity, or instructional value.These additional criteria could better characterize question quality.
  • Computational overhead: The proposal-rewriting-filtering-training loop requires additional computation compared with directly training on existing QA data, especially when applied iteratively.The overhead arises from repeatedly evolving and filtering questions before training.
  • Future directions: Batching, lightweight filtering, or more selective evolution strategies could reduce the framework’s computational overhead.Despite this cost, observed improvements in question quality support self-evolving visual questioners as a promising direction for stronger, more scalable questioning-oriented VLMs.

A Detailed Explanation of Questioning Capability Evaluation … B Detailed Prompts

The protocol evaluates visual question quality through individual-question perception and reasoning difficulty, plus question-set diversity. It also documents the prompts used for data generation and evaluation.

  • A Detailed Explanation of Questioning Capability Evaluation: The protocol assesses whether generated questions are visually grounded, challenging, and non-redundant at individual-question and question-set levels.Individual questions are evaluated through perception and reasoning difficulty, while question sets are evaluated for diversity.
  • A.1 Perception Difficulty: Perception difficulty measures the visual evidence that must be found, using Visual Search Difficulty and Visual Evidence Coverage.It ranges from salient or simple evidence to subtle, localized, fine-grained, or broadly distributed evidence.
  • A.1 Perception Difficulty: Visual Evidence Coverage scores questions higher when answering requires evidence from multiple objects, regions, attributes, scene cues, or the image’s overall arrangement.Coverage is independent of search difficulty, so broad evidence can be required even when each cue is easy to find.
  • A.2 Reasoning Difficulty: Reasoning difficulty measures the inference required after relevant visual evidence has been identified, using Visual Context Reasoning and Visual Spatial Reasoning.It distinguishes interpreting evidence from merely locating or inspecting it.
  • A.2 Reasoning Difficulty: Visual Context Reasoning scores questions higher when they require inferring scene context, object state, activity, function, intention, cause, or likely situation.Direct naming, reading visible attributes, and describing observable content receive lower scores.
  • A.2 Reasoning Difficulty: Visual Spatial Reasoning scores questions higher when they require comparing positions, orientations, distances, containment, occlusion, accessibility, layout, interactions, or multi-hop relations.Questions independent of spatial structure receive lower scores.
  • A.3 Questioning Diversity: Questioning Diversity is a question-set metric computed from average pairwise cosine distance among sampled, sentence-embedded questions for each image.Higher values indicate semantically less redundant questions covering more distinct image aspects, while lower values indicate repetition or duplicated targets.
  • B Detailed Prompts: Table 7 summarizes the prompts used for data generation and evaluation, with full prompt texts shown in Fig. 2–Fig. 7.This section provides the detailed prompts underlying both processes.

C Implementation Details · Question Proposal

The paper evaluates three instruction-tuned Qwen VLMs using a standardized self-evolution training setup, while its question-proposal procedure constructs challenging, visually grounded question–answer pairs through topic, subtopic, and visual-information constraints. Training retains 10,000 generated pairs and uses fixed optimization and evaluation configurations.

  • C Implementation Details: Experiments use Qwen2.5-VL-3B-Instruct, Qwen2.5-VL-7B-Instruct, and Qwen3VL-4B-Instruct from corresponding instruction-tuned checkpoints.All models are trained with standard supervised fine-tuning.
  • Question Proposal: The proposal process first samples an answerable topic uniformly, expands it into 60 subtopics, and uniformly selects one subtopic.The selected topic and subtopic are explicitly stated during the procedure.
  • Question Proposal: The model then writes a long question and answer requiring expertise in the selected subtopic without naming or reusing its words in the question.The question must rely on visual information, so its answer would change with that information.
  • Question Proposal: The pipeline includes rewriting with visual facts, rewriting with hard examples, and filtering stages for question refinement and selection.These stages are represented by the paper’s question-proposal figures and accompanying implementation description.
  • Question Proposal: 10,000 retained question-answer pairs train each model with batch size 2, AdamW, learning rate 2 × 10−5, cosine scheduling, warm-up ratio 0.1, and one epoch.The retained examples are produced by the self-evolution pipeline.
  • Question Proposal: Question-generation evaluation uses the same held-out images, prompt, and decoding configuration across models, with GPT-5.4 judging visual search difficulty, evidence coverage, context reasoning, and spatial reasoning.Set-level diversity uses Qwen/Qwen3-Embedding-4B to compute semantic distances among questions generated for the same image.
  • Question Proposal: Training and generation use official Qwen implementations with PyTorch and Hugging Face Transformers, while evaluation applies fixed prompts, rubrics, and decoding settings across compared methods.Diversity evaluation again uses Qwen/Qwen3-Embedding-4B and same-image question distances.

D Detailed QA Benchmark Results · E Human and QG Evaluation Alignment · E.1 Human Annotation Protocol

The detailed QA benchmarks evaluate five datasets and indicate that filtered self-generated data improves question-generation quality without broadly degrading downstream answering. Human evaluation samples 100 generated questions and scores four visual question-generation dimensions using the same rubrics as GPT-based evaluation.

  • D Detailed QA Benchmark Results: Table 8 reports full QA benchmark results for the main experiments and data-source ablation across five evaluation datasets.The datasets are CVBench-2D, CVBench-3D, SPARBench, VStar, and RealWorldQA.
  • D Detailed QA Benchmark Results: 1,438 samples comprise the CVBench-2D evaluation set, alongside 1,200 CVBench-3D samples.These sample counts are reported as part of the full QA benchmark evaluation.
  • D Detailed QA Benchmark Results: 7,207 SPARBench samples, 191 VStar samples, and 765 RealWorldQA samples complete the benchmark coverage.The reported results support evaluation across diverse visual question-answering datasets.
  • D Detailed QA Benchmark Results: Filtered self-generated data improves question-generation quality without causing broad degradation in downstream answering ability.This conclusion is supported by the full benchmark results and data-source ablation.
  • E Human and QG Evaluation Alignment: The human-GPT alignment study evaluates 100 randomly sampled generated visual questions using annotators familiar with vision-language tasks.Participation was voluntary and unpaid, and no private or sensitive information was collected.
  • E.1 Human Annotation Protocol: Annotators independently score visual search difficulty, visual evidence coverage, visual context reasoning, and visual spatial reasoning.They are shown the image, generated question, and scoring instructions for each dimension.
  • E.1 Human Annotation Protocol: The annotation interface and instruction screenshots are provided in Figure 8.The protocol focuses on whether questions are grounded in images and on the visual perception or reasoning required to answer them.

E.2 Human Alignment Results · F Comparison with LOVA3

GPT-based evaluation aligns positively with human judgments across all four question-generation dimensions, supporting its use as a scalable comparison proxy. Against LOVA3, the self-evolving models achieve stronger visual and reasoning-oriented question quality, with further gains in the second round.

  • E.2 Human Alignment Results: 0.662 Spearman correlation marks the strongest human–GPT alignment, observed for visual context reasoning.GPT-based scores show positive correlation with human judgments across all four dimensions.
  • E.2 Human Alignment Results: Positive correlations across all dimensions support GPT-based evaluation as a reasonable proxy for scalable question-generation comparison.The evaluator does not fully replace human judgment but captures meaningful trends in perceived question quality.
  • F Comparison with LOVA3: LOVA3-LLaVA-v1.5-7B is compared with the self-evolving Qwen2.5-VL-3B model, while LOVA3 relies on human annotations or stronger external models for instruction-tuning supervision.The comparison contextualizes the framework against a multimodal paradigm with question asking and VQA-triplet assessment capabilities.
  • F Comparison with LOVA3: Questions are compared under the same evaluation protocol across perception, reasoning, and diversity dimensions.This comparison focuses on visual question-generation quality.
  • F Comparison with LOVA3: 0.259 to 0.553 is the visual context reasoning increase from LOVA3 to the first-round model, which also improves visual search difficulty, visual evidence coverage, and visual spatial reasoning.The gains indicate questions requiring more contextual interpretation and spatial reasoning over image evidence.
  • F Comparison with LOVA3: 0.325 to 0.501 is the average QG score increase from LOVA3 to the second-round model, which achieves the highest average QG score.The result indicates more visual-centric and reasoning-demanding questions under the same evaluation protocol.

G Qualitative Analysis

Qualitative comparisons across generation rounds show that the self-evolving method progressively improves question generation. Questions shift from simple recognition or attribute-based prompts toward more visually grounded questions requiring stronger visual inspection and reasoning.

  • Generation-round progression: Across the same images, M0, M1, and M2 illustrate gradual improvement in question-generation capability.The comparison is presented in Figures 9 and 10.
  • Generation-round progression: The base model typically asks valid but shallow recognition or attribute-based questions requiring limited visual inspection or reasoning.Examples concern colors, object types, scene categories, shapes, and visible attributes.
  • Generation-round progression: After iterative generation and training, questions become more grounded in specific visual content and require stronger reasoning signals.The examples include cross-region reflection, scene mood, and spatial or layout reasoning.
Loading 2606.13929v1…