Source-linked AI summary
Groundbench: Multi-Resolution Polygon Grounding Exposes the Geometry Gap in Vision-Language Models
Zhonghan Bian, Zhenran Wang, Jinsong Li, Zhangyang Qi
TL;DR
Box-based grounding scores leave limited room to distinguish frontier systems, but rectangles discard object shape. Groundbench retargets the same image-expression-referent triples to exact-budget polygons and scores both region utility and legal completion. Its results expose an operational output-geometry gap across localisation, contour construction, serialisation, and topology.
Problem
Box-based referring-expression grounding leaves limited evidence about object shape and how well a model follows an object's contour.
Source: §1 · p. 1Method
Groundbench retargets 1,500 matched image-expression-referent triples to five fixed polygon budgets and scores filled-region utility and legal-polygon completion with fixed denominators.
Source: §1 · p. 2, §3.3 · p. 6Results
Gemini direct and box-derived polygons score 57.7 and 57.3 pooled IoU against the same contour target, while polygon performance peaks at moderate budgets and falls at N=64.
Source: §4.2 · p. 8Takeaways & Limitations
The paired utility and legality measures, alongside parsing, topology, preference, and tracing diagnostics, provide a common substrate for comparing vision-language and tool-assisted systems.
Source: §5 · p. 9The selection diagnostic is not verified instance detection because it uses no absolute overlap threshold, confidence ranking, margin, or cross-category matching.
Source: §3.4 · p. 7
Abstract
from arXiv · showhide
Bounding-box scores on RefCOCO-family grounding leave little room to distinguish frontier vision-language systems, yet boxes discard object shape. We introduce GroundingBench, a matched benchmark that re-targets the same 1,500 image-expression-referent triples to exact-N polygons at five vertex budgets. A fixed-denominator harness separately audits filled-region intersection over union (IoU) and legal-polygon completion. The strongest tested configuration reaches 88.2 box IoU and 97.1 accuracy at IoU >= .5 (Acc@.5), versus 57.7 and 69.2 for direct polygons; because these headline scores use different references, we also compare direct polygons with predicted boxes rasterised against the same contour target, obtaining 57.7 versus 57.3 when pooled. Performance is non-monotone in N and collapses at the densest budget, where legality failures compound residual geometric error. Qwen's thinking-setting contrast is the largest tested input-preserving configuration difference; under frozen templates, false spatial cues are more damaging than false colour cues, and target preference can remain high while contour tracing is poor. Alternate masks and a continuous-area scorer preserve the principal ordering. GroundingBench therefore measures an operational output-geometry gap spanning localisation, boundary construction, serialisation, and topology, rather than latent boundary perception alone.
1. Introduction
Groundbench reframes referring-expression grounding as exact-budget polygon output to expose what box metrics omit. Its matched design and diagnostics examine performance across five vertex budgets and distinguish multiple sources of output failure.
1. Introduction
- Box outputs omit object shape and boundary-following, while dense masks usually rely on specialist decoders or external segmentation modules.The paper positions fixed-budget polygons as an intermediate, textual representation for exposing shape through a general vision-language interface.Source: §1 · p. 1
- A fixed vertex budget tests whether added representational capacity helps despite longer outputs and more opportunities for duplicated points, crossings, shortcuts, and accumulated drift.Repeating each question across budgets exposes where added bandwidth stops helping.Source: §1 · p. 2
- Groundbench retargets 1,500 image-expression-referent triples to five exact-budget polygons, keeping each referent fixed while varying the answer surface.It uses 500 questions each from RefCOCO, RefCOCO+, and RefCOCOg, with shared UIDs for paired analysis.Source: §1 · p. 2
- The benchmark pairs filled-region utility with legal-polygon completion, scores unparseable responses as zero, and freezes composition metadata for controlled comparisons.Its contributions also include a frozen 13-condition diagnostic grid and checks under alternate masks and an independent backend.Source: §1 · pp. 2–3
- Across four hosted configurations, box localisation and polygon production remain separated, accuracy is non-monotone in N and collapses at 64, and false spatial cues hurt more than false colours.The Qwen thinking-setting contrast is the largest input-preserving configuration difference; truthful cues show no reliable gain.Source: §1 · p. 2
2. Related Work
Related work situates Groundbench between box grounding and specialist segmentation, and among methods that treat coordinates or polygons as output sequences. It also frames the benchmark as a matched retargeting of a familiar task and notes that polygon evaluation depends on explicit rasterisation choices.
2. Related Work
- Unlike specialist polygon and contour decoders, Groundbench audits general-purpose hosted LVLMs without task-specific training, using announced budgets and shared referents across resolutions.Prior work includes variable-length polygons for referring segmentation and vertex-sequence contour models.Source: §2 · p. 3
- Groundbench audits unmodified hosted APIs: fixed-N polygons make allocation, exact-length compliance, and topology auditable between short-box outputs and learned mask heads.The related work notes that neither boxes nor learned mask heads establish that chat decoders can maintain a long cyclic boundary.Source: §2 · p. 4
- Fixing N and reusing each referent across budgets separates representational capacity from delivery, requiring a dense, metrically scored geometric artefact rather than a categorical spatial judgement.This contrasts with spatial suites that report perceptual and relational weaknesses [Chen et al., 2024a; Fu et al., 2024; Liu et al., 2023b; Tong et al., 2024].Source: §2 · p. 4
- Polygon-region scores depend on choices for disconnected parts, self-intersections, clipping, and discrete integration, so the paper compares alternate masks and continuous-area intersection.These checks bound whether conclusions depend on one evaluator.Source: §2 · p. 4
- Groundbench retargets box-based referring-expression grounding to multi-resolution polygons on the same RefCOCO sources, isolating the output contract from changes in image, expression, or referent distribution.The section places this approach alongside benchmark retargeting examples including ImageNet-V2, SuperGLUE, MMLU-Pro, and AIME.Source: §2 · p. 4
3. The GROUNDBENCH Benchmark
Groundbench specifies exact-vertex polygon outputs and a frozen cohort, then scores filled regions, polygon legality, and selection-versus-tracing diagnostics. Its construction and scoring choices make the output requirements and evaluation boundaries explicit.
3.1. Task and Canonical Targets
- Models emit exactly N integer-coordinate vertices for an image and expression; filled scoring rasterises their clipped cyclic fill, while legality requires distinct in-range vertices and a simple positive-area polygon.Start point and traversal direction are unrestricted because filled-mask IoU is invariant to cyclic shifts and reversal.Source: §3.1 · p. 5
- Canonical targets use midpoint insertion for sparse contours and Visvalingam–Whyatt simplification for dense contours; largest-vertex-part selection affects 148/1,500 UIDs.The selected part averages 71.76% of the all-parts union and is also the largest-area part in 85.14% of affected UIDs.Source: §3.1 · p. 5
3.2. Cohort Construction and Rationale
- The cohort contains 1,500 unique image–referent pairs, 500 each from RefCOCO, RefCOCO+, and RefCOCOg, with distractor, complexity, scale, phrase-length, and difficulty metadata frozen.It includes 436 easy, 554 medium, and 510 hard questions; sampling preceded model outputs.Source: §3.2 · pp. 5–6
3.3. Scoring
- Fixed-denominator scoring counts every planned question, assigns zero to failed exact-length parses, and pairs overall scores with parseable-only scores; separate audits assess legality and alternate references.Construction fidelity is reported for each vertex budget, and headline scores use the uniform IoU2 view after up to two parse-failure-triggered retries.Source: §3.3 · pp. 6–7
3.4. Selection and Tracing Diagnostics
- Selection records whether target overlap exceeds every same-category distractor, while conditional tracing measures target IoU after that preference event; this diagnostic is not verified instance detection.Table 4 reports Sel as target preference, Con as target IoU conditional on Sel, and IoU with the fixed denominator.Source: §3.4 · p. 7, Table 4 · p. 7
4. Experiments
Across the matched polygon budgets, the experiments find a large box-to-polygon gap, a non-monotone budget effect, and substantial sensitivity to model configuration and input cues. The results also show that target preference, contour tracing, and legal completion are distinct aspects of performance.
4.1. Setup
- Four hosted configurations answer 1,500 questions at five budgets, with Qwen-off as a same-endpoint ablation; results are a purposive hosted-system snapshot, not an architecture-level comparison.Generations were collected 10–17 July 2026; headline IoU2 locks the first parseable output over one primary call plus two stateless retries.Source: §4.1 · p. 7
4.2. Main Results
- Pooled Gemini direct-polygon IoU is 57.7 versus 57.3 for predicted boxes rasterised to exact-N polygons against the same contour target.Headline box and contour IoU use different references, so their separation is contextual rather than a direct measure of latent geometric loss.Source: §4.2 · pp. 7–8
- At N=64, Gemini’s filled IoU is 37.7 while legal-polygon coverage/IoU fall to 7.5/4.2, and performance peaks at moderate budgets before collapsing.The decline persists when extra vertices only subdivide edges without changing the filled target, consistent with allocation, ordering, and topology burdens beyond parsing.Source: §4.2 · p. 8
- Acc@.5 preserves one ordering across the four main configurations and five budgets, but fixed IoU has a Qwen–Gemini inversion at N=64.Lower thresholds can produce further cross-budget changes, so one threshold at one budget underdetermines model quality.Source: §4.2 · p. 8
4.3. Ablation Study
- Qwen max-thinking exceeds thinking-off at every budget; removing the image is comparably damaging, while halving resolution or removing colour costs far less.Wrong spatial hints hurt more than wrong colours and combine sub-additively; truthful cues give no reliable gain.Source: Figure 3 · p. 8, §4.3 · p. 9
- Target preference stays high until the densest budget while conditional contour tracing remains a differentiator; alternate masks and a continuous-area backend shift five-budget means by at most 1.1 points without changing ordering.The cohort concentrates on same-class distractors within COCO/RefCOCO, limiting the benchmark to a controlled output audit rather than an architecture-wide claim.Source: §4.3 · p. 9
5. Conclusion
GROUNDBENCH retargets referring-expression grounding from boxes to fixed-budget polygons, with aligned targets and diagnostics for representation and delivery. Its paired utility and legality measures distinguish failure types as contours densify.
5. Conclusion
- GROUNDBENCH converts referring-expression grounding from boxes into fixed-budget polygon production for controlled study of representation and delivery.Aligned targets, deterministic scoring, frozen metadata, and parsing, topology, preference, and tracing diagnostics support that study.Source: §5 · p. 9
- Across matched budgets, richer contours eventually give way to parsing and topology failures, which diagnostics help localise.The curve can indicate whether losses arise in parsing, contour construction, or topology as the boundary densifies.Source: §5 · p. 9
- Paired region-utility and legality axes expose distinct failures, while coordinate emission is an audit stress test rather than a prescribed interface.Mask heads, tool calls, and compressed contours remain valid alternatives.Source: §5 · p. 9