PaperlayerView exact revision on arXiv

Source-linked AI summary

InstanceBench: Diagnosing Referential Reasoning and Target Identity in Referring Expression Segmentation

Yuchen Li, Shaoyang Zhou, Yiran Wang, Ruiyi Deng, Haoyu Wang, Ziru Wei, Zhen Zhao, Luping Zhou

arXiv:2610.09478v1cs.CV

TL;DR

Standard RES evaluation offers limited evidence about whether models preserve a target across distinct valid grounding paths, and mask overlap conflates target selection with mask generation. InstanceBench measures identity-aware performance across logic-controlled expression sets; its evaluation finds a gap between the strongest checkpoint’s expression-level overlap and set-level success, with target selection the main bottleneck.

  • Problem

    Standard RES evaluation does not systematically organize expressions by referential logic, test target preservation across valid grounding paths, or separate target-selection from mask-generation errors.

    Source: §1 · pp. 1–2

  • Method

    InstanceBench uses fixed-target expression sets, a taxonomy of referential logics, and identity-aware metrics to assess target retention and distinguish selection from mask-generation failures.

    Source: §1 · pp. 2–3, §2.2 · p. 4

  • Results

    The strongest checkpoint reaches 67.1% expression-level mIoU but only 59.6% All@0.7, and failure decomposition identifies target selection as the dominant bottleneck in key relational and compositional failures.

    Source: §1 · p. 3

  • Takeaways & Limitations

    InstanceBench maps capability across referential logics and shows that identity-aware performance responds to matched logic-aware supervision.

    Source: §1 · p. 3, §5 · p. 10

    InstanceBench is a controlled diagnostic benchmark, not an estimate of RES performance under natural distributions.

    Source: §5 · p. 10

Abstract

from arXiv · show

Referring Expression Segmentation (RES) links natural-language descriptions to pixel-level object masks. Yet standard evaluation provides limited insight into instance-level referential reasoning: it does not systematically distinguish referential logics, test target preservation across valid grounding paths, or separate target-selection from mask-generation errors. We introduce InstanceBench, an instance-centered diagnostic benchmark comprising 6,194 images, 9,264 target instances, and 25,077 human-verified expressions. Each target-centric expression set (TCES) fixes the image and target mask while pairing a minimal expression with a same-target variant that uses another valid cue or grounding path. A compact referential-logic taxonomy spans direct target evidence, same-class selection, relational and compositional grounding, and exclusion, while logic-critical construction suppresses simpler shortcuts. Identity-aware metrics measure target retention and set-level success while separating selection from mask-generation errors. Across 22 native-mask RES checkpoints from 18 model families, the strongest checkpoint reaches 67.1% mIoU but only 59.6% All@0.7. Controlled interventions confirm language sensitivity, while failure decomposition identifies target selection rather than mask decoding as the main bottleneck. On a controlled training subset, matched supervision improves identity-aware performance, showing that the diagnosed capability responds to targeted supervision. Collectively, InstanceBench supports a measure-diagnose-improve cycle: measuring target consistency across grounding paths, localizing failure sources, and evaluating targeted interventions.

1 INTRODUCTION

Standard RES evaluation offers limited insight into whether models execute different forms of instance-level reasoning and preserve a target across valid grounding paths. InstanceBench evaluates these capabilities and separates target-selection from mask-generation errors.

1 INTRODUCTION

  • Prior benchmarks broaden expression difficulty and diagnostic coverage, but generally score expressions independently rather than systematically testing whether distinct grounding paths preserve one target’s identity.Standard mask overlap also conflates target selection with mask generation, leaving referential reliability largely unmeasured.Source: §1 · pp. 1–2
  • InstanceBench’s TCESs hold the image and target mask fixed while pairing a minimal expression with a same-target variant using a different valid cue or grounding path.Its nine-category taxonomy covers direct target evidence, same-class selection, relations, multi-anchor composition, and exclusion; shortcut-controlled construction makes the annotated logic necessary for disambiguation.Source: §1 · pp. 2–3
  • 67.1% expression-level mIoU but only 59.6% All@0.7 is reached by the strongest checkpoint; target selection is the dominant bottleneck in key relational and compositional failures.Across 22 native-mask checkpoints from 18 model families, performance varies sharply across referential logics, and controlled interventions establish language sensitivity.Source: §1 · p. 3
  • Identity-aware and set-level metrics test target preservation across grounding paths and distinguish target-selection from mask-generation failures.These diagnostics complement conventional mask-overlap metrics and organize evaluation around referential reasoning rather than isolated expression–mask pairs.Source: §1 · pp. 2–3

2 THE INSTANCEBENCH DATASET

InstanceBench combines multi-instance images from COCO and OpenImages with newly constructed, human-verified expressions. Its design fixes a target within each expression set, varies the grounding path, and verifies that the intended logic is necessary and unambiguous.

2.1 DATA SOURCES AND SPLITS

  • The benchmark contains 6,194 images, 9,264 target instances, and 25,077 retained human-verified expressions across 221 normalized categories.Training, validation, and test splits contain 3,299/895/2,000 images, 4,896/1,450/2,918 TCESs, and 14,083/3,624/7,370 expressions, respectively; GE4-qualified source images are supplemented with OpenImages to broaden category coverage.Source: §2.1 · p. 3

2.2 DATASET DESIGN AND CONSTRUCTION

  • A TCES fixes one image and target mask while grouping valid expressions that all refer to that target.Its minimal expression uses only necessary identifying cues, while the same-target variant uses another valid cue or grounding path; dense diagnostic sets add paraphrases that preserve logic and path.Source: §2.2 · pp. 3–4
  • The taxonomy defines five evidence structures and nine canonical categories, assigned only when the corresponding reasoning operation is necessary to identify the target uniquely.The categories cover target evidence, same-class selection, single-anchor relations, multi-anchor composition, and negation or exclusion; appearance and action remain auxiliary metadata.Source: §2.2 · p. 4
  • For each GE4-qualified target, a quota-aware scheduler selects two feasible logic–cue specifications while favoring logic diversity, then generates a minimal expression and same-target variant.The image, target identity, and mask remain fixed, and every expression must be logic-critical and shortcut-controlled; GPT-5.5 is given the taxonomy and validity contract rather than choosing the logic freely.Source: §2.2 · p. 4
  • Two different annotators serially review every proposal for mask quality, referential correctness, uniqueness, naturalness, logic necessity, and taxonomy assignment.Of 9,885 resolved targets, 9,264 (93.7%) were retained; a blinded audit found 91.7% agreement on five-family assignments and 95.7% agreement on selected target identity.Source: §2.2 · p. 5

3 EVALUATION PROTOCOL

InstanceBench evaluates expression-level mask quality, TCES-level reliability, and target-identity preservation. Its protocol separates identity selection from mask quality and measures consistency across alternative expressions and grounding paths.

3 EVALUATION PROTOCOL

  • The protocol reports expression-level mIoU, oIoU, and Precision@τ, alongside TCES-level reliability and target-identity metrics.The headline metrics are mIoU, All@0.7, and All-target-ID.Source: §3 · p. 5
  • All@τ requires every evaluated expression in a TCES to succeed, while worst-path IoU exposes the weakest grounding path.The protocol uses τ ∈{0.7, 0.9}.Source: §3 · p. 5
  • Identity evaluation assigns each predicted mask to the most compatible available annotated instance and credits only assignment to the TCES target.It evaluates TCESs with at least two available category-instance masks; assignments require both an absolute score threshold and a margin over the runner-up, otherwise identity is unresolved.Source: §3 · p. 5
  • All-target-ID measures set-level target preservation, while IDRet and TM@τ measure companion-expression identity retention and joint target-and-mask retention.Agreement on the same wrong instance receives no credit; TM@τ additionally requires both predictions to meet the mask threshold.Source: §3 · p. 6
  • Normalized consistency compares same-logic paraphrases with alternative-path variants against their marginal success rates; negative ΔNCI indicates weaker identity coupling after path changes.NCI is zero under marginal independence and one at the maximum co-success permitted by the marginals.Source: §3 · p. 6
  • Failure localization separates identity errors, including confirmed switches to another same-category instance, from correct-identity/poor-mask errors.Conditional mask IoU is computed only when both predictions are assigned to the annotated target, isolating mask stability after correct target selection.Source: §3 · p. 6

4 EXPERIMENTS

Across the InstanceBench test set, strong expression-level mask overlap does not guarantee reliable target selection across expressions or grounding paths. The experiments identify logic-specific and target-identity failures, test language sensitivity, and show improved identity-aware performance under matched targeted supervision.

4.1 EXPERIMENTAL SETUP

  • The main evaluation uses 2,000 test images, 2,918 TCESs, and 5,836 core expressions across 18 model families and 22 released checkpoints.Comparisons are clustered by TCES, with 95% bootstrap confidence intervals for central matched comparisons.Source: §4.1 · pp. 6–7

4.2 OVERALL PERFORMANCE: HIGH MIOU DOES NOT IMPLY SET-LEVEL RELIABILITY

  • 67.14% mIoU for SAMTok contrasts with 59.60% All@.7, showing that strong expression-level overlap does not ensure both core expressions succeed at IoU 0.7.The core test set contains 2,918 TCESs and 5,836 expressions.Source: Table 2 · p. 7, §4.2 · p. 7
  • Qwen3.8-27B reaches 69.83% mIoU and 63.48% All@.7 in box-to-mask evaluation, while Gemini-3.8-Flash reaches 65.17% All-target-ID under oracle candidate selection.These diagnostic protocols use a separate segmentation backend or oracle candidate masks, so they are not directly comparable to native-mask RES evaluations.Source: §4.2 · p. 7

4.3 WHICH REFERENTIAL LOGICS CHALLENGE CURRENT MODELS?

  • Ordinal selection, exclusion, and relational composition are weaker than direct target-centric evidence and extremal descriptions, with ordinal gaps persisting under matched controls.Ordinal mIoU is lower than extremal mIoU by 10.27–25.52 points across four models; nested grounding is also below matched spatial controls for three models.Source: §4.3 · pp. 7–8, Figure 4 · p. 8

4.4 SAME-TARGET CONSISTENCY AND FAILURE LOCALIZATION

  • Alternative-path retention is 78.50–89.18%, below paraphrase retention of 94.31–97.55%; target-identity errors account for 73.45–80.79% of failures on single-anchor and compositional expressions.When both predictions select the annotated target, their mask IoU is 90.16–98.73%, identifying target selection as the main bottleneck in these failures.Source: §4.4 · p. 8, Table 3 · p. 8

4.7 QUALITATIVE DIAGNOSIS

  • Under a fixed target, qualitative examples show grounding-path changes redirecting predictions to another same-category instance, while another case retains the instance but yields an incomplete mask.These examples illustrate the diagnosed mechanisms rather than their prevalence.Source: Figure 5 · p. 9, §4.7 · p. 9

4.5 LANGUAGE SENSITIVITY IS NOT REFERENTIAL RELIABILITY

  • Replacing expressions with category names reduces mIoU by 36.95–44.91 points, while replacing them with descriptions of another same-category instance shifts 90.29–94.23% of predictions toward that target.The interventions indicate language sensitivity, but the paper distinguishes that sensitivity from reliable referential reasoning and target preservation.Source: §4.5 · p. 9

4.6 THE DIAGNOSED CAPABILITY RESPONDS TO SUPERVISION

  • On a matched COCO-derived training subset, logic-aware supervision improves LAVT’s mIoU, All@.7, and All-target-ID by 5.00, 7.07, and 7.55 points, respectively.DETRIS benefits most from a 50/50 supervision mixture, with gains of 1.78, 2.55, and 4.34 points on those metrics.Source: §4.6 · p. 9

5 CONCLUSION

InstanceBench evaluates whether different valid grounding paths preserve target identity, and its conclusion identifies target selection as the main source of errors. It complements existing benchmarks by fixing the image and target while changing the grounding path, with a taxonomy organized by the evidence needed to identify the target.

5 CONCLUSION

  • Average mIoU overstates whole-TCES reliability, and errors arise mainly from target selection rather than mask generation.Counterfactual text interventions show language sensitivity but failures to preserve identity across valid grounding paths; the controlled-subset study shows this capability responds to supervision.Source: §5 · p. 10

A EXPANDED RELATED WORK

  • Unlike benchmarks focused on compositional grounding, broader output types, or counterfactual stress tests, InstanceBench holds the image and target fixed while varying valid grounding evidence.This makes its focus same-target reliability across grounding paths rather than changes in the requested output.Source: A EXPANDED RELATED WORK · p. 13

B TAXONOMY ANNOTATION GUIDE

  • Same-class set selection identifies a target by ordering, comparing, or grouping same-category candidates, while single-anchor relations require resolving one anchor and applying a target–anchor relation.An extremal description is distinct from scene-frame spatial language, and merely mentioning another object does not make an expression relational.Source: B TAXONOMY ANNOTATION GUIDE · p. 13
  • Multi-anchor and nested grounding use multiple jointly necessary constraints or a hierarchy, while target-centric evidence uses visible target properties and exclusion eliminates candidates by a necessary condition.The taxonomy treats exclusion as a cross-cutting logic and one canonical evaluation category.Source: B TAXONOMY ANNOTATION GUIDE · pp. 13–14, Table 4 · p. 14

C INDEPENDENT QUALITY-AUDIT PROTOCOL

The independent audit samples quota-balanced TCESs, obtains independent judgments, and resolves disagreements through adjudication. The annotation workflow then separates logic feasibility analysis from generation of two expressions for the fixed target.

C INDEPENDENT QUALITY-AUDIT PROTOCOL

  • The audit independently reviews 300 expressions from 150 image-unique TCESs, balanced across source images and canonical logics, with target identity and related labels hidden.Annotators judge target selection, logic, uniqueness, logic criticality, and shortcut control; a third adjudicator resolves disagreements, with confidence intervals clustered by TCES.Source: C INDEPENDENT QUALITY-AUDIT PROTOCOL · pp. 14–15
  • The completed audit reports 91.7% five-family agreement and 89.3% nine-category agreement, with annotators selecting the same target identity for 95.7% of expressions.Their selections match the fixed target on 95.3% and 98.3% of expressions, respectively.Source: C INDEPENDENT QUALITY-AUDIT PROTOCOL · p. 15

D GPT ANNOTATION PROMPTS

  • The annotation process uses analysis and generation calls separated by quota-aware assignment, and the implementation labels referential logic with the field name operation.Generation produces a minimal expression and a same-target variant; both must preserve the target, reject same-class competitors, and realize distinct assigned logic rather than a paraphrase.Source: D GPT ANNOTATION PROMPTS · p. 15, §D.2 · pp. 18–19

D.2 STAGE 2: EXPRESSION GENERATION

  • Generation checks require each operation to be necessary and shortcut-controlled, with valid anchors and expressions that uniquely identify the fixed target.The stage returns no expressions if either fails a required check, rather than weakening or relabeling the assigned operation.Source: §D.2 · pp. 18–19

D.1 STAGE 1: FEASIBILITY ANALYSIS

Stage 1 proposes feasible referential logics for a fixed target and selects a compact menu for expression generation. Its checks prioritize distinct logic families and reject operations that do not uniquely and necessarily identify the target.

D.1 STAGE 1: FEASIBILITY ANALYSIS

  • The analysis stage considers feasible operations for the fixed target and normally returns two or three operations from different high-level families when available.Four may be returned when all are independently strong and natural; one is allowed only when no reliable cross-family pair exists.Source: §D.1 · p. 15
  • Each candidate operation must preserve the target, uniquely reject same-class competitors, and be necessary to the identification problem without an easier unary shortcut.Anchors must be visible, independently identifiable, and non-circular; candidates failing mandatory checks are omitted rather than invented to improve balance.Source: §D.1 · pp. 15–16
  • The taxonomy covers same-class selection, single- and multi-anchor relations, nested grounding, direct target evidence, and negation or exclusion.Same-class selection includes ordinal, extremal, relative, and subgroup-based forms; exclusion must uniquely select one present target.Source: §D.1 · pp. 16–17

E TRAINING-REGIME AUDIT

The training-regime audit describes heterogeneous downstream supervision for evaluated checkpoints and cautions that their comparison is a transfer-performance comparison. The accompanying experimental-details section collects protocols, full results, dataset composition, and robustness analyses omitted from the main paper.

E TRAINING-REGIME AUDIT

  • Evaluated checkpoints span universal, R+, R, CoHD, and RefChess training regimes, with different mixtures of downstream supervision beyond classic RefCOCO-family RES.The regimes include generic or reasoning segmentation, grounded conversation, region question answering, interactive segmentation, video data, generalized RES, or pretrained components without task-specific training.Source: E TRAINING-REGIME AUDIT · p. 20
  • Because the supervision regimes differ, Table 2 is a transfer-performance comparison; controlled conclusions rely on the matched-supervision study.Table 5 summarizes the downstream supervision associated with representative evaluated checkpoints.Source: E TRAINING-REGIME AUDIT · p. 20

F ADDITIONAL EXPERIMENTAL DETAILS AND DATASET STATISTICS

  • The additional experimental details section documents protocols and supplies complete logic profiles, interventions, dense-track and source analyses, dataset composition, and robustness checks.It collects material omitted from the main paper for space.Source: F ADDITIONAL EXPERIMENTAL DETAILS AND… · p. 20

F.1 GENERAL-PURPOSE MLLM EVALUATION

The evaluation uses two protocols to examine general-purpose MLLMs on InstanceBench: direct localization and oracle candidate-ID selection. Both are diagnostic setups rather than directly comparable native-mask RES evaluations.

F.1 GENERAL-PURPOSE MLLM EVALUATION

  • Direct localization asks a model for one bounding box from the image and core expression, then converts that box to a mask with a shared frozen SAM3 backend.Invalid boxes receive zero overlap; valid boxes use SAM 3’s highest-confidence proposal, upsampled and thresholded at 0.5.Source: §F.1 · pp. 20–21, Table 6 · p. 21
  • The direct-localization pipeline is diagnostic and not directly comparable with native-mask RES models, while candidate-ID evaluation isolates selection within the annotated instance space.Candidate-ID results are an upper-context diagnostic rather than the standard RES input protocol.Source: Table 7 · p. 21, Table 6 · p. 21
  • The oracle candidate-ID protocol supplies the scene, expression, category, and valid ID range, and scores whether the model returns the target’s withheld ID.It is restricted to TID and uses identical yellow boxes with deterministic, target-independent IDs.Source: §F.1 · p. 21, Table 7 · p. 21

F.2 MATCHED-SUPERVISION RESULTS

The supplementary analyses examine matched supervision, referential-logic profiles, language interventions, and dense-track results. They report persistent difficulty with compositional references and no reliable repair from simple inference-time prompting changes.

F.3 COMPLETE REFERENTIAL-LOGIC PROFILES

  • Direct target evidence is consistently easier than nested, exclusion, and other composition-heavy references; matched analyses find a robust ordinal penalty and a generally persistent, smaller nested-grounding penalty.The multi-anchor contrast is weaker, with intervals often including zero.Source: §F.3 · p. 22, Table 9 · p. 22

F.4 CONTROLLED LANGUAGE INTERVENTIONS

  • Donor mIoU reaches 76.60–80.61 after same-image, same-category text swaps, while explicit-logic prompting, deliberation, and oracle same-class context produce small, model-dependent effects that do not reliably repair the failure.The intervention mIoU changes range from −2.05 to +2.34 points across the reported prompting conditions.Source: §F.4 · pp. 22–23, Table 10 · p. 23

F.5 DENSE DIAGNOSTIC TRACKS AND SOURCE TRANSFER

  • In dense tracks, alternative-path and exclusion expressions are less reliable than paraphrases, but optional-track mIoUs are descriptive because tracks cover different TCES subsets.The source-specific mIoU and All@0.7 comparisons also reflect differences in category and logic composition.Source: §F.5 · p. 23, Table 12 · p. 24, Table 13 · p. 24

F.6 SOURCE-STRATIFIED REFERENTIAL-LOGIC ROBUSTNESS

Source-stratified analyses find that the relative difficulty of referential logics is strongly preserved across COCO and OpenImages, despite lower absolute performance on OpenImages. The supplementary composition views describe coverage across sources and logic categories.

F.6 SOURCE-STRATIFIED REFERENTIAL-LOGIC ROBUSTNESS

  • Per-model COCO–OpenImages category-ranking correlations range from 0.750 to 0.950, with a median of 0.833; ordinal is harder than extremal and nested harder than single-anchor for every model in both sources.Target-centric evidence and extremal descriptions remain among the easier categories, though their first-place ranking can vary.Source: §F.6 · pp. 25–26, Table 15 · p. 26
  • Identity errors account for 66.46–78.79% of analyzed COCO failures and 73.50–81.34% of OpenImages failures; target-selection errors still dominate the OpenImages nested slice.The correct-ID/poor-mask share is modestly larger for OpenImages nested expressions than COCO’s, 26.50% versus 21.21%.Source: §F.6 · p. 26, Table 16 · p. 26

F.7 SUPPLEMENTARY DATASET COMPOSITION

  • The release composition views show complementary COCO and OpenImages image distributions and coverage of all nine canonical referential logics.Figure 8 summarizes target-category and logic coverage; Figure 9 shows test-set source mix and grounding-path transitions.Source: Figure 7 · p. 25, §F.7 · p. 26

F.8 FAILURE LOCALIZATION AND STATISTICAL ROBUSTNESS

Failure decomposition attributes most relational and compositional failures to identity errors, while the reporting notes clarify how uncertainty and threshold sensitivity are assessed.

F.8 FAILURE LOCALIZATION AND STATISTICAL ROBUSTNESS

  • The sensitivity analysis varies identity-assignment score and margin thresholds, reporting core-pair All-target-ID over 14,335 model–TCES pairs and failure categories over 5,289 low-IoU predictions.Failure columns cover single-anchor, multi-anchor, and nested categories.Source: Table 18 · p. 27
  • Identity errors account for 73.28–76.73% of relational and compositional failures, with confirmed same-class switches alone accounting for 53.73–58.04%.Same-class switches are a subset of identity errors.Source: Table 17 · p. 26, §F.8 · p. 28
  • Confidence intervals resample TCESs to preserve within-mask dependence, and smaller logic-profile categories are interpreted through their uncertainty.Logic-profile counts range from 113 relative-selection to 1,830 single-anchor expressions.Source: §F.8 · p. 28
Loading 2610.09478v1…