VG-Refiner: Towards Tool-Refined Referring Grounded Reasoning via Agentic Reinforcement Learning

• Existing tool-integrated reasoning often overlooks unreliable tool outputs, which can mislead models into hallucinated reasoning, particularly in referring expression comprehension.The passage connects this issue to downstream spatial grounding and counting tasks.Source: §1 · p. 2
• In the motivating example, REVPT fails to detect an irrational tool result even though the original Qwen2.5-VL-7B locates the true object.Source: §1 · p. 2
• VG-Refiner uses a think–rethink process to analyze tool feedback and decide whether to accept or refine it, while also following reliable tool results.Its second stage checks consistency among visual evidence, the text query, and tool predictions.Source: §1 · p. 2
• The paper introduces a refinement reward, the PiTER evaluation protocol, and two metrics for measuring refinement capability and quality.PiTER injects tool results into the system prompt and uses JSON grounding outputs for fair comparison.Source: §1 · p. 2
• VG-Refiner outperforms SOTA methods on RefCOCO/+/g accuracy and surpasses Qwen2.5-VL-32B in refinement ability while preserving pretrained general capabilities.The model is trained with a small amount of task-specific data.Source: §1 · p. 2

• Existing tool-integrated visual reasoning uses tools for visual attention, text extraction, code-based reasoning, and visual perception, but detection-tool noise can disrupt referring comprehension.Examples include iterative zoom-in and region-of-interest selection, OCR, code interpretation, depth estimation, and detection.Source: §2 · p. 2
• REC research includes supervised grounding datasets and RL methods using verbal chain-of-thought, while proposal-based methods reason over detected boxes to find targets.The passage states that no existing methods use agentic RL for REC with external tool references.Source: §2 · p. 2
• VG-Refiner’s overview combines IoU-sensitive rewards for correcting or accepting tool outputs with GRPO rollouts and KL constraints to a frozen reference model.The model queries a referring visual toolkit after its think process.Source: Figure 2 · p. 3
• The paper defines a weak model as detection-capable but not fine-tuned on referring datasets; expert models such as EVF-SAM and InstanceVG use BEIT-3 encoders.Source: §2 · p. 3
• The paper identifies overlooked erroneous tool outputs in REC as the motivation for a tool-refined referring grounded reasoning paradigm.Source: §2 · p. 3

3.1. Tool-refined Referring Grounded Reasoning

• VG-Refiner’s TrRGR paradigm explicitly checks external-tool outputs, correcting unreliable results while optimizing reliable ones for higher IoU.This contrasts with tool-integrated methods that invoke tools without deeper reasoning about unreliable results and verbal-CoT models without external tool collaboration.Source: §3.1 · p. 3
• The think–rethink policy first forms a broad object hypothesis, then uses tool feedback to re-evaluate the prediction and make targeted corrections.The expert model returns a detection result from the image and referring phrase; the policy produces the final refined result.Source: §3.1 · p. 4
• VG-Refiner uses a strong REC tool at inference, but a weak tool when benchmarking its ability to refine suboptimal predictions.A strong tool has SOTA performance on mainstream REC benchmarks; a weak tool has not been fine-tuned on referring-expression data.Source: §3.1 · p. 4

3.2. Agentic RL Training

• GRPO trains VG-Refiner with multiple reasoning-trajectory rollouts and rewards for structured output and tool-feedback refinement.The format reward checks the required think, rethink, and answer tags.Source: §3.2 · p. 4
• The refinement reward combines confirmation and refinement components to adapt tool use to feedback quality, assigning 1 for correcting wrong tool outputs and 0.5 when the tool is already correct.The coarse piecewise reward is intended to stabilize training and avoid reward hacking from noisy IoU fluctuations.Source: §3.2 · pp. 4–5

3.3. Refinement Ability Evaluation

• PiTER provides a shared single-stage prompt that injects tool results, removes intermediate verbal reasoning, and requires JSON grounding predictions for fair comparison.The prompt is used across model types, regardless of their original reasoning or tool-integration paradigms.Source: §3.3 · p. 5
• CCR measures successful refinements on tool-failed cases, while NSRI measures IoU improvement relative to the maximum possible gain.NSRI is averaged over tool-failed cases; positive normalized change represents improvement.Source: §3.3 · pp. 5–6

4.1. Experimental Settings

• Experiments cover RefCOCO/+/g and LISA-Grounding for grounding, plus five visual-QA benchmarks; NSRIw and CCR assess refinement on incorrect-tool samples.Training uses 9K RefCOCOg samples and one epoch, with EVF-SAM and non-finetuned GD-T tool outputs.Source: §4.1 · p. 6

4.2. Main Results

• VG-Refiner reaches state-of-the-art RefCOCO-series performance, comparable to 72B-scale models, and consistently beats Qwen2.5-VL-7B and expert-tool outputs.The model integrates EVF-SAM feedback with its own reasoning in the rethink stage.Source: §4.2 · pp. 6–7
• It improves across five test sets with Rex-Omni and UNINEXT-H despite EVF-SAM-only training, and outperforms 7B/32B models under weak tools while avoiding REVPT’s negative strong-tool NSRIw.In zero-shot LISA-Grounding evaluation, it also surpasses the Qwen2.5-VL-7B baseline when EVF-SAM is an unseen, weak tool.Source: §4.2 · pp. 7–8

4.4. Ablation Study

• The refinement reward conditions verifiable scores on tool correctness, giving full reward for correcting bad tools and 0.5 for marginally refining correct ones; results gain across benchmarks.The authors report higher overall accuracy and NSRIw with this hierarchical reward.Source: Table 6 · p. 7, §4.4 · pp. 8–9
• Ablations show second-stage tool feedback improves grounding, while a further rethink stage supports additional refinement and self-correction.Table 7 reports the think–rethink mechanism ablation in accuracy.Source: Table 7 · p. 8, §4.4 · p. 9

4.3. Quality Assessment

• Qualitatively, rethink uses tool context to resolve boundary ambiguity and can reject completely incorrect predictions across visualized tool-error cases.The paper illustrates three representative tool-induced error types; the first two often occur with EVF-SAM and the third with non-finetuned Grounding DINO-T.Source: Figure 4 · p. 7, §4.3 · p. 8

4.4. Ablation Study

• With baseline Qwen predictions as PiTER tool feedback, VG-Refiner corrects suboptimal outputs and gains consistently, unlike Qwen2.5-VL-7B.Table 8 evaluates this self-tool setting on RefCOCOg test subset.Source: Table 8 · p. 8, §4.4 · p. 9
• Despite task-specific refinement training, VG-Refiner preserves general visual-QA ability and gains slightly on some benchmarks.Table 5 reports general QA evaluation across challenging benchmarks.Source: Table 5 · p. 7, §4.4 · p. 8

5. Conclusion

• VG-Refiner combines a refinement reward and two-stage think–rethink framework to improve external-tool feedback refinement while retaining general visual-question-answering capabilities.The model is trained with a limited amount of data.Source: §5 · p. 9

Supplementary Material

• Figure 6 presents common errors in referring expression comprehension and a RES-model output mask for “A lady is walking away from the surfer.”Source: Figure 6 · p. 12
• Table 9 reports visual-grounding evaluation results using Acc@0.5 on RefCOCO, RefCOCO+, and RefCOCOg.Source: Table 9 · p. 12

6. The Choice of Tools

• Referring expert tools may mislocalize objects, produce imprecise boundaries, or miss objects; even strong tools can occasionally select the wrong object.Source: §6 · p. 12
• EVF-SAM masks are converted to boxes using their extreme pixel coordinates, but ambiguous multi-object masks or artifacts can make those boxes imprecise.Source: §6 · p. 12
• Non-finetuned Grounding DINO-T serves as a weak tool because it struggles with complex, long referring expressions while handling simple category-name queries reasonably well.Its average mean accuracy is around 0.4 on RefCOCO/+/g test sets.Source: §6 · p. 12
• Table 10 compares models under self-tool conditions on RefCOCO+ testA and testB using PiTER.Source: Table 10 · p. 12

7. More Benchmark Results

• REVPT's overall benchmark performance remains relatively low, with format failures, insufficiently specific REC inputs, and predictions that track tool outputs rather than refine them.The authors identify these issues as consistent with their earlier observations.Source: §7 · p. 12
• VG-Refiner achieves performance comparable to a 32B-scale model on additional test sets using tool outputs from the original baseline.The self-improvement experiments use JSON-formatted tool results generated by reproduced Qwen2.5-VL-7B and also evaluate outputs from the original baseline.Source: §7 · p. 12
• Follow Correct Rate measures how reliably the refiner preserves correct tool predictions when its IoU remains nearly unchanged.The definition is based on the IoU relationship between tool predictions and refiner outputs.Source: §7 · pp. 12–13
• Worsen Rate measures the overall fraction of dataset samples in which the refiner degrades the tool output.The displayed formula expresses this fraction as a percentage, with N denoting the total number of samples.Source: §7 · pp. 12–13
• VG-Refiner more faithfully follows reliable tool outputs while avoiding further degradation than Qwen2.5-VL-32B under PiTER.Table 11 reports FCR and WR results across multiple referring datasets, but the supplied passages do not give the table values.Source: §7 · p. 13, Table 11 · p. 13

8. Training Details

• The refinement reward remains above 0.5 during training, indicating effective correction of incorrect tool outputs and receipt of the corresponding refinement reward.The tool's overall IoU hovers around 0.6, providing an appropriate level of difficulty and contributing to stable training dynamics.Source: §8 · p. 13
• The format reward stays near 1 throughout training, indicating reliable adherence to the required structured output format.The reward curves report mean values across the batch.Source: §8 · p. 13
• The 9k-sample training set is balanced between EVF-SAM outputs and Grounding DINO T outputs.Half of the data is randomly sampled from each source.Source: §8 · p. 13

9. PiTER protocol Analysis

• PiTER removes intermediate reasoning, tool-calling, and iterative zooming, which are typically part of TiVR methods.The authors describe this as a purposeful design choice for their experimental setup.Source: §9 · p. 13
• Because evaluated tool-use models share the Qwen2.5-VL-7B pretrained backbone, PiTER aims to isolate refinement ability added by fine-tuning rather than compare full agentic pipelines.The shared models therefore have identical foundational visual-language capabilities and architectural priors.Source: §9 · p. 13
• A single-stage, tool-injected inference format standardizes interaction so performance differences reflect learned refinement mechanisms rather than procedural reasoning frameworks.The authors say native multi-step interaction could confound comparisons through differences in external scaffolding.Source: §9 · pp. 13–14

10. More Case Study

• Figure 8 presents additional cases in which VG-Refiner successfully corrects tool failures.The reported cases are visual examples of correction.Source: §10 · p. 14
• Most of the additional correction cases originate from non-finetuned Grounding DINO T.The passage characterizes this tool as relatively weak in modeling spatial relationships and object-attribute associations.Source: §10 · p. 14
• The cited weakness in those Grounding DINO T outputs concerns spatial relationships and object-attribute associations.These are the specific capabilities noted in connection with the examples.Source: §10 · p. 14