Self-Evolving Visual Questioner

1 Introduction

• Prior VQG methods rely on static human-curated or external supervision (Mi et al., 2024; Shen et al., 2024; Zhao et al., 2024; Xie et al., 2025b; Zhang et al., 2025a), limiting question diversity, grounding, and reasoning complexity.Continuously collecting annotations is expensive and difficult to scale, while unconstrained self-generated questions can collapse into repetitive, low-information behavior.Source: §1 · pp. 1–2
• The framework uses a VLM to propose questions under multiple visual intents, rewrite and filter them for grounding, informativeness, reasoning, and diversity, then train in QG and QA modes and repeat.It builds supervision from unlabeled images without human annotations, external teacher models, or auxiliary reward models.Source: §1 · p. 2
• After two self-evolving rounds, average QG score increases by ∼82% relative to the initial base model, while QA accuracy remains stable and improves on several benchmarks.The proposal-and-filter mechanism maintains exploration diversity and visual-centric reasoning during self-evolution.Source: §1 · p. 2
• Ablations show generated data provides a more effective small-budget training signal than directly sampled source annotations.Experiments and ablations cover multiple models, text and image sources, and filtering strategies.Source: §1 · p. 2
• The agentic evaluation protocol assesses visual-question quality across perception, reasoning, and diversity, enabling fine-grained analysis beyond aggregate QA accuracy.Generated questions better search for informative visual evidence, cover broader image content, and show stronger contextual and spatial reasoning.Source: §1 · p. 2

2.1 Overview

• The framework proposes, rewrites, and filters questions, then uses retained pairs for dual-format training and reuses the adapted model as proposer in an iterative loop.Filtering considers answerability, visual grounding, and perception and reasoning difficulty; QA supervision anchors answering behavior while QG supervision teaches question and answer generation.Source: §2.1 · p. 2, Figure 1 · p. 3

2.2 Problem Formulation

• Visual question generation conditions informative questions on an image and optional controls such as question type, answer style, or reasoning intent.These intents encourage broader coverage of the question space for each image; without control information, the model generates questions directly from the image.Source: §2.2 · p. 3

2.3 Agentic Questioning Capability Evaluation Protocol

• The evaluation measures perception through visual search and evidence coverage, reasoning through contextual and spatial interpretation, and question-set diversity through redundancy among questions for each image.A GPT-based judge scores individual questions conditioned on the image, while semantic distances from sentence embeddings measure diversity within each image’s question set.Source: §2.3 · p. 3

3.1 Question Proposal

• The current model proposes image-grounded, answerable questions across varied intents and generates answers to form candidate question-answer triples.Intents include recognition, comparison, spatial relations, scene understanding, and grounded reasoning.Source: §3.1 · p. 4

3.2 Question Rewriting

• The evolving model Mt proposes questions, while the initial model M0 rewrites them toward harder, more visual-centric directions to diversify refinement.Rewrite directions include visual inspection difficulty, evidence grounding, contextual reasoning, and spatial reasoning.Source: §3.2 · p. 4

3.3 Question Filtering

• A model-based filter retains rewritten pairs that remain visually answerable and grounded and improve perception or reasoning difficulty, discarding ambiguous, weakly grounded, unanswerable, or unimproved candidates.The retained candidates form the training dataset for the next stage.Source: §3.3 · p. 4

3.4.1 Dual-Format Supervision

• Dual-format supervision trains question generation from images alone while using image-question inputs for answer generation to preserve downstream VQA ability.The two supervision sets are mixed for supervised fine-tuning.Source: §3.4.1 · pp. 4–5

3.4.2 Iterative Self-Improvement

• Iterative self-improvement reuses each adapted model as the next proposer while M0 rewrites and filters its proposals, and the combined components target stronger question generation with competitive downstream QA.Rewriting expands candidates toward harder, more visual-centric questions; filtering removes invalid or non-improving generations, and dual-format supervision anchors answering behavior.Source: §3.4.2 · p. 5

4.1 Experimental Setup

• Evaluation covers three VLM backbones, 100 held-out CVBench images for five QG dimensions, and four QA benchmarks; default training uses 10K QA+QG examples.QG uses five questions per image, with GPT-5.4 judging individual-question dimensions and Qwen3-Embedding-4B measuring within-image diversity.Source: §4.1 · p. 5

4.2 Main Results

• Self-evolution consistently improves all five QG dimensions across evaluated backbones, including strong Qwen3VL-4B, without degrading downstream QA.With 10K self-supervision examples, the dual-format setup improves QG without shifting the model away from its answering behavior.Source: §4.2 · pp. 5–6, Table 1 · p. 6
• Second-round adaptation further improves all five QG dimensions over the first round by reusing the adapted model as proposer and refining its questions.The framework reuses the same unlabeled image pool, progressively shifting generated questions toward broader coverage, stronger grounding, and greater perception and reasoning difficulty.Source: §4.2 · p. 6, Table 1 · p. 6

4.3.1 Effect of Supervision Format

• QA+QG training improves QG average over QA-only across all three backbones while achieving the best or comparable QA performance; QG-only can reduce QA.QG-only generally improves QG over QA-only, while QA-format supervision helps preserve answering ability.Source: §4.3.1 · p. 6, Table 2 · p. 7

4.3.2 Effect of Self-Evolution and Filtering

• Adding filtering to rewriting raises Search 0.353 to 0.433, Coverage 0.520 to 0.547, Context 0.507 to 0.553, Spatial 0.293 to 0.367, and Diversity 0.320 to 0.350.Direct self-generated supervision already improves four QG dimensions over the base model, and rewriting further strengthens all five; filtering selects stronger candidates from the expanded pool.Source: §4.3.2 · p. 6, Table 3 · p. 7

4.3.3 Effect of Self-Supervision Data

• Under a matched 10K budget, self-supervision achieves the strongest overall QG profile and best QA average, while mixing it with Original SAT remains weaker than self-supervision alone.Original SAT improves visual search but has lower average QA accuracy and weaker reasoning and diversity scores than self-supervision.Source: §4.3.3 · p. 7, Table 4 · p. 8

4.4 Do Better Questions Help Supervision?

• 63.32% versus 61.90% average QA accuracy is achieved using improved-model questions rather than base-model questions, with GPT-generated answers held constant.Accuracy rises from 69.25% to 75.58% on CVBench-3D; the improved-question setting also performs better on CVBench-2D, VStar, and RWQA.Source: §4.4 · p. 7, Table 5 · p. 8

5.1 Visual Question Generation

• VQG uses VQA-style, LLM-generated, and controllable data, but fixed datasets, generators, and control spaces limit autonomous development of visual-questioning capability (Mi et al., 2024; Shen et al., 2024; Vedd et al., 2022; Zhao et al., 2024; Suwono et al., 2023; Zhang et al., 2025a; Fang et al., 2024; Uehara and Harada, 2023; Xie et al., 2025a,b).These approaches can improve answerability, grounding, diversity, or difficulty control.Source: §5.1 · p. 8
• VQG evaluation often uses reference-based BLEU, METEOR, and CIDEr, while LLM judges mainly assess general multimodal response or text-domain question quality rather than fine-grained image-grounded questioning (Mi et al., 2024; Vedd et al., 2022; Xie et al., 2025a,b; Chen et al., 2024; Noorbakhsh et al., 2025; Yao et al., 2025).Source: §5.1 · p. 8

5.2 Post-Training for VLMs

• Prior VLM self-improvement uses self-generated signals mainly for answering capabilities, whereas this work post-trains the visual questioner itself while preserving answering ability (Yuan et al., 2024; Deng et al., 2024; Cheng et al., 2025; Wang et al., 2024; Pan et al., 2026).Collecting high-quality human, curated-dataset, or stronger-model instruction data is costly and difficult to scale.Source: §5.2 · p. 8

• Experiments show substantial question-generation improvements while largely preserving question-answering performance; ablations validate each component and the downstream value of improved questions.Source: §6 · p. 8
• The framework creates visual-centric, diverse, reasoning-oriented supervision from unlabeled images through question proposal, rewriting, filtering, and dual-format QA/QG training.Source: §6 · p. 8
• The paper introduces an evaluation protocol covering perception, reasoning, and diversity.Source: §6 · p. 8

7 Limitations

• The filter targets answerability and perception/reasoning difficulty, but finer criteria could better capture visual grounding, ambiguity, and instructional value.Source: §7 · p. 8
• The proposal-rewriting-filtering-training loop adds computation over direct training on existing QA data, especially iteratively; batching, lightweight filtering, or selective evolution could reduce the overhead.Source: §7 · pp. 8–9
• Observed improvements in question quality suggest self-evolving visual questioners are promising for building stronger, more scalable questioning-oriented VLMs.Source: §7 · p. 9

A Detailed Explanation of Questioning Capability Evaluation

• The protocol evaluates individual questions for perception and reasoning difficulty, then assesses question sets for complementary rather than redundant supervision.Perception concerns finding visual evidence; reasoning concerns interpreting it.Source: A Detailed Explanation of Questioning… · p. 11

A.1 Perception Difficulty

• Perception difficulty comprises visual search difficulty—how hard evidence is to locate—and visual evidence coverage—how broadly a question draws on image evidence.Search scores rise for subtle, small, occluded, peripheral, fine-grained, comparative, or countable evidence; coverage scores rise when multiple objects, regions, attributes, scene cues, or arrangements matter, independently of search difficulty.Source: §A.1 · p. 11

A.2 Reasoning Difficulty

• Reasoning difficulty measures the inference needed after evidence is identified and is decomposed into visual context reasoning and visual spatial reasoning.This distinguishes interpreting evidence from finding and inspecting it.Source: §A.2 · p. 11
• Visual context reasoning scores questions higher when they require inferring scene context, object state, activity, function, intention, cause, or likely situation beyond direct recognition or description.Questions answerable by naming objects, reading visible attributes, or describing directly observable content receive lower scores.Source: §A.2 · p. 11
• Visual spatial reasoning scores questions higher when they require reasoning about relations such as position, orientation, distance, containment, occlusion, accessibility, layout, interaction, or multi-hop spatial structure.Questions not dependent on spatial structure receive lower scores.Source: §A.2 · p. 11

A.3 Questioning Diversity

• Questioning diversity is measured across questions for each image by repeatedly sampling questions, encoding them with a sentence-embedding model, normalizing embeddings, and deriving average pairwise cosine distance.The procedure computes average pairwise cosine similarity within each sampled set before defining diversity as the corresponding average cosine distance.Source: §A.3 · p. 11
• Higher diversity scores indicate semantically less redundant questions covering distinct image aspects, while lower scores indicate repeated templates, near-paraphrases, or duplicated answer targets.This set-level measure complements individual-question scores by assessing diversity across generated supervision.Source: §A.3 · p. 11

B Detailed Prompts

• The detailed prompts for data generation and evaluation are summarized in Table 7, with full prompt texts shown in Fig. 2–Fig. 7.Source: B Detailed Prompts · p. 11

C Implementation Details

• Models are fine-tuned from instruction-tuned checkpoints on 10,000 retained question-answer pairs for one epoch, using batch size 2 and AdamW.The learning rate is 2 × 10^-5 with a cosine scheduler and 0.1 warm-up ratio.Source: C Implementation Details · pp. 11–12
• The experiments use Qwen2.5-VL-3B-Instruct, Qwen2.5-VL-7B-Instruct, and Qwen3VL-4B-Instruct.Source: C Implementation Details · p. 11

Question Proposal

• The proposal prompt produces an initial question-answer pair, while the section also identifies rewriting with visual facts and hard examples.Table 7 summarizes prompts used in the self-evolving questioner framework and agentic visual-questioning evaluation.Source: Figure 2 · p. 12, C Implementation Details · p. 12, Table 7 · p. 12
• A training-ready set of 10,000 self-supervision pairs takes about 24 GPU hours to generate with 16-way parallel generation; one-epoch fine-tuning takes about 1–3 GPU hours per model.A full benchmark evaluation takes about 4–7 GPU hours per checkpoint; most runs use one GPU, with RTX A5000s and A100s used across the specified models.Source: C Implementation Details · p. 12
• Question-generation evaluation uses GPT-5.4 to judge visual-search difficulty, evidence coverage, context reasoning, and spatial reasoning, with scores averaged across questions or images by dimension.The evaluation uses a held-out image set and shared generation prompt and decoding configuration across compared models.Source: C Implementation Details · pp. 12–13, Table 6 · p. 12
• Set-level diversity is measured with Qwen/Qwen3-Embedding-4B semantic distances among questions generated for the same image; downstream QA uses benchmark protocols and accuracy averaged over three runs.VLMEvalKit is used for downstream QA, with inference settings fixed across checkpoints.Source: C Implementation Details · p. 13

D Detailed QA Benchmark Results

• Filtered self-generated data improves question-generation quality without causing broad degradation in downstream answering ability.Table 8 reports the full main-experiment and data-source-ablation QA benchmark results.Source: D Detailed QA Benchmark Results · p. 13
• The QA evaluation covers CVBench-2D (1,438 samples), CVBench-3D (1,200), SPARBench (7,207), VStar (191), and RealWorldQA (765).The results are reported in Table 8.Source: D Detailed QA Benchmark Results · p. 13

E.1 Human Annotation Protocol

• The alignment study samples 100 generated visual questions and uses internal student researchers familiar with vision-language tasks as voluntary, unpaid annotators.The task scores generated questions using the same rubrics as the automatic evaluator.Source: §E.1 · p. 13
• Annotators independently score visual search difficulty, visual evidence coverage, visual context reasoning, and visual spatial reasoning after seeing each image, question, and scoring instructions.The annotation interface and instruction screenshots are shown in Figure 8.Source: §E.1 · p. 13
• Annotators assess image grounding and the visual perception or reasoning needed to answer, without identifying people, inferring protected attributes, or making decisions about them.Questions come from public research datasets and are screened for clearly offensive or unsafe content before annotation.Source: §E.1 · p. 13
• The annotation collects no private or sensitive information, and its results are used only for agreement measurement and reported in aggregate correlation statistics.Participation is voluntary, and annotators provide no personal information.Source: §E.1 · p. 13

E.2 Human Alignment Results

• 0.662 is the Spearman correlation between human and GPT scores for visual context reasoning, the strongest alignment among the four dimensions.Visual evidence coverage has the lowest correlation, consistent with its need for fine-grained judgments about relevant objects, regions, and image evidence.Source: §E.2 · p. 13, Table 9 · p. 14
• Positive correlations across all dimensions support GPT-based evaluation as a reasonable proxy for scalable question-generation evaluation, though it cannot fully replace human judgment.Source: §E.2 · p. 13

F Comparison with LOVA3

• The comparison uses LOVA3-LLaVA-v1.5-7B against our Qwen2.5-VL-3B results; LOVA3 uses existing instruction-tuning supervision, while ours is self-generated and self-refined from unlabeled images.LOVA3 supervision comes from human annotations or stronger external models such as GPT; our framework uses neither human annotations nor external teacher models.Source: F Comparison with LOVA3 · p. 13
• Questions are compared with LOVA3 under the same QG protocol, which evaluates perception, reasoning, and diversity.Source: F Comparison with LOVA3 · pp. 13–14
• Our first-round model improves over LOVA3 on all four QG dimensions; visual context reasoning rises from 0.259 to 0.553.The largest gains are on reasoning dimensions, including visual context reasoning and visual spatial reasoning.Source: F Comparison with LOVA3 · p. 14
• 0.501 is the second-round model’s average QG score, up from LOVA3’s 0.325.The results indicate more visual-centric and reasoning-demanding questions under the same evaluation protocol.Source: F Comparison with LOVA3 · p. 14

Qualitative Analysis

• Across rounds, questions shift from shallow recognition toward specific, visually grounded reasoning, with M2 further improving specificity and relation-centered reasoning.The reported progression is consistent with gains in visual search difficulty, evidence coverage, visual context reasoning, spatial reasoning, and diversity.Source: G Qualitative Analysis · p. 14, Figure 9 · p. 20, Figure 10 · p. 21
• Examples in Figure 9 show questions involving cross-region reflection, relative-size comparison, depth and layout understanding, and fine-grained counting with exclusion.Source: G Qualitative Analysis · p. 14, Figure 9 · p. 20
• Figure 10 illustrates questions requiring scene-mood, spatial and layout, activity, and human-engagement reasoning.Source: Figure 10 · p. 21