Source-linked AI summary
Draft and Refine with Visual Experts
Sungheon Jeong, Ryozo Masukawa, Jihong Park, Sanggeon Yun, Wenjun Huang, Hanning Chen, Mahdi Imani, Mohsen Imani
TL;DR
LVLMs can rely on linguistic priors rather than visual evidence, and there is no quantitative measure of how much they use visual information during reasoning. DnR measures question-conditioned visual utilization and uses it to guide expert-assisted refinement. Experiments report consistent accuracy gains and reduced hallucination across VQA and captioning benchmarks.
Problem
LVLMs may rely on linguistic priors rather than visual evidence, and their use of visual information during reasoning lacks a quantitative measure.
Source: §1 · p. 1Method
DnR builds a query-conditioned relevance map, measures visual dependence through relevance-guided probabilistic masking, and uses utilization to guide expert-assisted refinement.
Source: §1 · p. 1, §3.1 · p. 3, §3.2.1 · p. 3Results
Takeaways & Limitations
DnR offers a measurable criterion for evaluating and leveraging visual experts in multimodal reasoning.
Source: §1 · p. 1, §6 · p. 9The current hybrid masking is heuristic, and its effectiveness varies with dataset, masking ratio, rendering style, and expert setup.
Source: §5 · pp. 8–9
Abstract
from arXiv · showhide
While recent Large Vision-Language Models (LVLMs) exhibit strong multimodal reasoning abilities, they often produce ungrounded or hallucinated responses because they rely too heavily on linguistic priors instead of visual evidence. This limitation highlights the absence of a quantitative measure of how much these models actually use visual information during reasoning. We propose Draft and Refine (DnR), an agent framework driven by a question-conditioned utilization metric. The metric quantifies the model's reliance on visual evidence by first constructing a query-conditioned relevance map to localize question-specific cues and then measuring dependence through relevance-guided probabilistic masking. Guided by this metric, the DnR agent refines its initial draft using targeted feedback from external visual experts. Each expert's output (such as boxes or masks) is rendered as visual cues on the image, and the model is re-queried to select the response that yields the largest improvement in utilization. This process strengthens visual grounding without retraining or architectural changes. Experiments across VQA and captioning benchmarks show consistent accuracy gains and reduced hallucination, demonstrating that measuring visual utilization provides a principled path toward more interpretable and evidence-driven multimodal agent systems. Code is available at https://github.com/EavnJeong/Draft-and-Refine-with-Visual-Experts.
1. Introduction
The introduction argues that expert coordination should respond to a model’s use of question-relevant visual evidence rather than linguistic confidence. DnR measures that utilization and uses rendered expert feedback to refine predictions without retraining.
1. Introduction
- Language-driven expert selection inherits language-model biases, while learned coordination requires costly, inflexible joint optimization across experts and tasks.The introduction asks whether a model can choose when and which expert to invoke based on its perceptual needs.Source: §1 · p. 1
- DnR measures utilization of visual evidence relevant to the particular question, rather than maximizing visual dependence globally.Its utilization criterion guides adaptive expert choice based on the model’s perceptual behavior rather than linguistic priors or heavy joint supervision.Source: §1 · p. 1
- A relevance-based perturbation process quantifies reliance on visual information without additional supervision.The measure addresses the absence of an explicit label or metric for assessing visual dependence.Source: §1 · p. 1
- Rendered expert outputs highlight essential image regions and suppress irrelevant content, integrating diverse experts without retraining.The experts provide additional evidence to complement perceptual gaps.Source: §1 · p. 2
- DnR consistently improves VQA and captioning performance, correlates strongly with task accuracy, and substantially reduces hallucination.The introduction notes that rendering styles and parameters require dataset- and model-specific tuning.Source: §1 · p. 2
2. Related Work
Prior work spans multimodal models, tool-using agents, and approaches to hallucination and grounding. The related-work section contrasts these methods’ reliance on language, heuristics, or global corrections with DnR’s measurable, query-conditioned use of visual evidence.
Related work
- Multimodal models perform strongly on visual tasks, but language-driven reasoning can yield unsupported content and limited grounding on fine-grained or localized benchmarks.Prior evaluations motivate explicitly assessing, verifying, and refining visual evidence.Source: §2 · p. 2
- Tool-augmented systems use programmatic agents, expert coordinators, and multimodal planners, but often lack measurable visual-evidence criteria for expert invocation.Their invocation commonly depends on textual outputs, embedding similarities, or predefined heuristics.Source: §2 · p. 2
- Hallucination-mitigation methods include supervision, retrieval, and uncertainty-driven selection, while some post-hoc or global amplification approaches lack query conditioning.The paper contrasts these with a relevance map that links visual importance to question semantics for selective refinement.Source: §2 · p. 2
3. Draft and Refine with Visual Experts
DnR drafts an answer, measures its use of question-relevant visual evidence, and uses that measure to guide refinement with visual experts. Its utilization score combines sensitivity to relevant regions with stability to irrelevant ones.
3. Draft and Refine with Visual Experts
- DnR refines an LVLM’s draft by adding structured visual evidence from selected experts and re-querying the model.Expert outputs are rendered onto the image as visual cues the existing vision encoder can process, without architectural modification.Source: §3 · p. 2, §3.3 · p. 5
3.1. Query-Conditioned Relevance Map
- A language model converts the question into concrete visual queries, which a CLIP-based grounder localizes and averages into a relevance map.The map highlights image regions most informative for answering the question.Source: §3.1 · p. 3
3.2.1. Relevance-based probabilistic masking
- Regions are sampled for Top-k or Bottom-k masking according to normalized relevance scores, perturbing approximately a ρ fraction of the image.An adaptive factor weights the two strategies using relevance-map entropy and contrast; sharper maps favor Top-k masking, while diffuse maps favor Bottom-k masking.Source: §3.2.1 · pp. 3–4
3.2.2. Semantic deviation measurement.
- A semantic encoder compares masked-image predictions with the original: Top-k deviation measures dependence on relevant regions, while Bottom-k deviation measures stability to irrelevant-region masking.The two deviations are combined using the adaptive factor.Source: §3.2.2 · pp. 4–5, §3.2.3 · p. 5
3.2.3. Utilization score.
- The utilization score balances dependence on question-critical regions against stability when irrelevant regions are perturbed; higher scores indicate more visually grounded reasoning.The adaptive factor balances the Top-k and Bottom-k contributions.Source: §3.2.3 · p. 5
3.3. Expert Selection and Incorporation
- DnR renders each expert’s structured output onto the original image, re-queries the LVLM, and selects the expert yielding the largest utilization gain.If no expert improves utilization over the draft, the system skips refinement.Source: §3.3 · p. 5
3.4. Learned Expert Selection
- A learned selector predicts the expert with the largest utilization gain to avoid evaluating every candidate; Table 1 covers 16 benchmarks and reports Draft/DnR performance.The table caption also identifies revision rates, correction-versus-degradation transitions, and correlations; it does not provide cell values in the supplied passage.Source: §3.4 · p. 5, Table 1 · p. 6
4. Experiments
The experiments test whether utilization-guided single-step selection affects accuracy, hallucination, and evidence-grounded reasoning. They use four visual experts and evaluate how DnR performs across multimodal reasoning settings.
4. Experiments
- Experiments test whether selecting the candidate that maximizes Uq improves accuracy, reduces hallucination, and promotes evidence-grounded reasoning.DnR performs a single-step selection.Source: §4 · p. 6
- The evaluation uses four visual experts from distinct backbone families to provide complementary cues.GroundingDINO, SAM, DepthAnything, and mDETR are the selected experts.Source: §4 · p. 6
- Utilization is computed with 16 stochastic masks per image, using Gumbel-k sampling and masking ratios set by mode.Top-k uses 0.25, Bottom-k uses 0.75, and Hybrid adjusts the ratio adaptively.Source: §4 · p. 6
4.1. Comprehensive Evaluation
DnR’s effects vary with baseline visual grounding and task demands, with reported improvements across VQA, captioning, visual reasoning, knowledge-based VQA, and comprehensive benchmarks. The evaluation also tests correlations between utilization scores and accuracy gains.
4.1. Comprehensive Evaluation
- The evaluation spans five multimodal reasoning categories and tests Pearson and Spearman correlations between utilization scores and accuracy gains.All experiments use GRAY rendering and Hybrid masking.Source: §4.1 · p. 6
4.1.1. VQA
- Across VQA models, weaker-grounded systems revise more and gain more, while stronger systems revise less but still improve.IDEFICS and MiniGPT-v2 show revision rates of ≈8.3–11.3% and gains of +1.4–3.0%; LLaVA 1.6, CogVLM, and Qwen2.5-VL show rates of ≈3.0–4.6% and gains of +1.1–1.5%.Source: §4.1.1 · p. 6
- VQA gains scale with visual demand: perception-focused benchmarks improve more than text-centric tasks.VQAv2 and VizWiz show gains of 1.8–2.7%, versus 1.4–1.6% on OCR-VQA and TextVQA; IDEFICS–VizWiz is an exception tied to answering previously avoided cases.Source: §4.1.1 · p. 6
4.1.2. Image Captioning
- In image captioning, DnR improves CIDEr across models, with larger gains for PaliGemma and LLaVA 1.6 than for generic-description models.Gains are +2.0–4.4 for IDEFICS and InstructBLIP and +6.6–12.1 for PaliGemma and LLaVA 1.6; NoCaps gains are +1.7–5.3.Source: §4.1.2 · pp. 6–7
4.1.3. Visual Reasoning
- On visual reasoning benchmarks, DnR improves semantic and spatial reasoning, with gains even when revisions are infrequent.VCR gains reach +2.0–5.5 for visually weaker models and +0.1–1.0 for stronger models; VSR revisions occur in <1% of cases but yield +0.5–2.5% gains.Source: §4.1.3 · p. 7
4.1.4. Knowledge-based VQA
- On knowledge-based VQA, DnR reinforces perceptual grounding but cannot replace missing conceptual knowledge.Accuracy gains are +0.5–2.0%; A-OKVQA shows near-zero utilization–accuracy correlation (≤0.2) and minimal accuracy change.Source: §4.1.4 · p. 7
4.1.5. Comprehensive Benchmark
- On comprehensive benchmarks, DnR makes few revisions (≤2%) and yields small but consistent gains, primarily refining consistency once grounding is stable.Utilization–accuracy correlations are 0.3–0.5; weaker-grounded IDEFICS and MiniGPT-v2 gain mainly on perception-oriented MME metrics (+31–39).Source: §4.1.5 · p. 7
4.2. Hallucination
DnR reduces hallucinations across the evaluated benchmarks and increases grounding and correctness on average. The reported changes also show that hallucinated responses may shift toward misperception as well as grounded responses.
4.2. Hallucination
- The evaluation uses HaloQuest, MMHal-Bench, VizWiz, and COCO Caption, with responses categorized as Hallucination, Misperception, Grounded, or Correct.The benchmarks were chosen for detailed hallucination annotations rather than binary labels.Source: §4.2 · p. 7
- DnR consistently reduced hallucinations across all benchmarks by 1–9 pp; InstructBLIP fell 6.78 pp (26.7%) and Qwen2.5-VL 5.96 pp (35.0%), while LLaVA 1.6 had the largest proportional reduction (29.1%).Source: §4.2 · p. 7
- Average grounding increases by 0.6–5.9 pp, while average correctness increases by 0.5–2 pp across benchmarks.CogVLM gains +3.94 pp (+33.6%) in grounding; PaliGemma gains +3.46 pp (+7.4%) in correctness.Source: §4.2 · pp. 7–8
- Hallucinated responses are largely redirected toward Misperception and Grounded categories, with some samples shifting only to Misperception.A VizWiz exception occurs for IDEFICS, which becomes less conservative about answering “unanswerable” cases.Source: §4.2 · pp. 7–8
4.3. Policy-Driven Expert Selection
The learned selector uses the utilization metric Uq to choose an expert or initial draft instead of exhaustively evaluating experts. It substantially reduces computational cost with minimal average performance difference across the reported benchmarks.
4.3. Policy-Driven Expert Selection
- The three-layer MLP selector predicts which expert or initial draft yields the highest Uq, replacing exhaustive expert evaluation.Occasional Refine-for-Draft swaps typically had negligible impact when the responses matched and the utilization difference was below ϵ.Source: §4.3 · p. 8
- Using the query set Q rather than the raw question q improves selector performance through stronger alignment with visual representations.Source: §4.3 · p. 8
- Approximately 70% lower computational cost with an average performance difference of around -0.4 pp across four benchmarks, excluding MME’s distinct scoring scale.COCO had the smallest cost reduction (-52.7%) because it inherently involves more experts during generation, leaving less room for pruning.Source: §4.3 · p. 8
- Guiding expert selection by Uq turns visual utilization into an objective for deciding when and how experts should intervene, supporting larger and more diverse expert pools.Source: §4.3 · p. 8
5. Discussion
Performance gains correlate linearly with utilization, but utilization levels and changes differ across datasets, architectures, and inputs. The discussion identifies rendering and hybrid masking as sensitive to the setup and points to adaptive policies as a possible alternative.
5. Discussion
- Performance gains show a clear linear correlation with utilization, while Uq and its change ∆Uq vary across datasets, architectures, and individual inputs.Some models show high but stable Uq, whereas others have larger ∆Uq shifts linked to stronger gains.Source: §5 · p. 8
- Hybrid masking remains a heuristic whose effectiveness varies with dataset, masking ratio ρ, rendering style, and expert setup.The authors describe rendering as DnR’s most flexible yet sensitive component.Source: §5 · p. 8
- Adaptive policies could extend rendering beyond a fixed single-step heuristic to multistep refinement and combine experts’ textual outputs for visual and linguistic consistency.Source: §5 · p. 9
6. Conclusion
The conclusion presents DnR as a scalable agent framework for quantifying and improving LVLMs’ use of visual evidence. Across diverse benchmarks, it reports accuracy gains, reduced hallucination, and clearer attribution of visual reasoning.
6. Conclusion
- DnR coordinates multiple visual experts to quantify and improve how LVLMs use visual evidence.Source: §6 · p. 9
- Experiments across diverse benchmarks demonstrate consistent accuracy gains, reduced hallucination, and clearer attribution of visual reasoning.Source: §6 · p. 9
- The authors position DnR as a principled, scalable foundation for systematically integrating and expanding expert-driven multimodal reasoning.Source: §6 · p. 9