PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision-Language Grounding as Bidirectional Concept Correspondence

Jieyu Zhang, Ziqi Gao, Luke Zettlemoyer, Ranjay Krishna

arXiv:2608.07886v1cs.CVcs.AIcs.CL

TL;DR

Most grounding methods localize prespecified text, leaving the discovery of visually referential spans and their image correspondences unaddressed. ConCor-1 formulates this as bidirectional concept correspondence and consistently outperforms baselines, improving correspondence F1 by 48% on long captions and 29% on zero-shot LVIS.

  • Problem

    Existing grounding tasks typically localize prespecified text, rather than jointly identifying visually referential spans, image entities, and their correspondences.

  • Method

    ConCor-1 uses learned bridge tokens to jointly predict text masks, image masks, and correspondence validity from paired images and text.

  • Results

    ConCor-1 consistently performs best across image-caption and image-category grounding, improving correspondence F1 by 48% on COCONut-PanCap over the strongest baseline.

  • Takeaways & Limitations

    Bidirectional concept correspondence provides a unified output space for caption grounding, referring-expression grounding, and open-vocabulary grounding.

  • Takeaways & Limitations

    ConCor-1’s mask quality remains behind specialized detection and segmentation models, and its training depends on jointly annotated correspondence supervision.

Abstract

from arXiv · show

Vision-language grounding connects language to visual content, yet most existing formulations reduce grounding to a unidirectional localization problem: given a prespecified text phrase or category name, identify the corresponding image region. This setup assumes that the relevant linguistic unit is already known, overlooking a more basic challenge in grounded communication: determining which parts of the text are visually referential and how they correspond to entities in the image. We formulate grounding as $\textit{bidirectional concept correspondence}$ over an image-text pair. Given an image and its paired text, the goal is to recover all correspondences between visually referential text spans and instance-level image segments, without assuming that the relevant text spans are provided. This formulation unifies common grounding tasks, including phrase grounding, referring expression grounding, and open-vocabulary detection, by treating text segmentation, image segmentation, and cross-modal alignment as a single correspondence prediction problem. To address this task, we introduce $\textbf{ConCor-1}$, a grounding model built on top of a pretrained vision-language model. It uses learnable $\textit{bridge tokens}$ to represent candidate image-text correspondences and predicts, for each token, a text mask, an image mask, and a correspondence presence score. To train and evaluate this task, we convert diverse grounding and segmentation datasets into a unified correspondence format. Experiments show that $\textbf{ConCor-1}$ consistently outperforms baselines, improving correspondence F1 by 48% on the long-caption dataset and by 29% on zero-shot LVIS, where the large category list serves as the text input.

1 Introduction

The paper reframes vision-language grounding as bidirectional concept correspondence: jointly identifying visually referential text spans and their corresponding instance-level image segments. ConCor-1 implements this task with learnable bridge tokens and lightweight prediction heads, and is evaluated across unified grounding settings.

  • Problem formulation: Grounding is reformulated from unidirectional language-to-image localization into recovering all correspondences between referential text spans and instance-level image masks.The formulation predicts text masks marking mentions of the same visual entity and corresponding image masks.
  • Problem formulation: The formulation unifies phrase grounding, referring expression grounding, and open-vocabulary detection and segmentation as special cases.Phrase grounding assumes spans are specified, category-name input yields open-vocabulary grounding, and referring-expression grounding additionally addresses contextual entities.
  • ConCor-1: ConCor-1 uses learnable bridge tokens contextualized over image and text tokens to represent candidate correspondences.Similarity with vision and text tokens produces image segments and text masks, while a presence head predicts whether each correspondence is valid.
  • ConCor-1: ConCor-1 adds lightweight heads for image segmentation, text segmentation, and correspondence-presence prediction on top of a pretrained vision-language backbone.The design supports reasoning over long, compositional text and makes text segmentation part of the output space.
  • Evaluation: ConCor-1 is trained and evaluated after converting diverse grounding, detection, and segmentation datasets into a unified correspondence format.Evaluation covers image-caption grounding and image-category grounding, with the model consistently performing best across both settings.

2 Related work

Prior vision-language grounding formulations vary in how linguistic queries are specified and what visual outputs are required. Related models span referring-expression and phrase grounding methods, detector-style and promptable segmentation systems, and autoregressive vision-language models.

  • Vision-language grounding: Grounding formulations differ mainly in the specified linguistic query and required visual output.
  • Vision-language grounding: Referring expression comprehension localizes a target object or region from a short phrase, while phrase grounding aligns multiple caption phrases with image regions.
  • Grounding models: Prior grounding systems include detector-style methods, generalized or promptable segmentation models, and autoregressive vision-language systems.Examples include MDETR, GLIP, Grounding DINO, X-Decoder, SEEM, SAM3, Florence-2, Molmo, LISA, and GLaMM.

3 Bidirectional concept correspondence

This section defines grounding as bidirectional correspondence between visually referential text segments and instance-level image segments, without prespecified text spans. ConCor-1 operationalizes this task with bridge tokens that jointly predict text masks, image masks, and correspondence validity.

  • 3 Bidirectional concept correspondence: Grounding outputs correspondence pairs linking grounded text segments to corresponding image segments for a paired image and text.Text segments are represented as character-level binary masks, allowing discontinuous or multi-span expressions.
  • 3 Bidirectional concept correspondence: The model must resolve entity mentions and coreference, identify candidate objects, and determine which mentions and objects correspond, while allowing unmatched mentions or objects.This formulation also supports evaluating caption groundedness and text-to-image generation faithfulness.
  • 3 Bidirectional concept correspondence: For each bridge token, ConCor-1 predicts text and image masks plus a presence score indicating whether the token represents a valid correspondence.Training jointly supervises text masks, visual masks, and correspondence validity using BCE, Dice, and point-sampled visual-mask losses.
  • 3 Bidirectional concept correspondence: The task requires structured correspondence prediction beyond local text–pixel affinities, including compositional referring structures and fine-grained instance masks.Challenges include modifiers, repeated mentions, pronouns, and grouping tokens and pixels into referential masks.
  • 3 Bidirectional concept correspondence: ConCor-1 uses a pretrained vision-language backbone and learned bridge tokens, with each bridge token representing a candidate image–text correspondence.The bridge tokens jointly attend to visual and textual tokens to hypothesize alignments between image regions and text spans.

4 Experiments

Experiments evaluate bidirectional concept correspondence across image-caption and image-category settings using segmentation, correspondence, ablation, and qualitative analyses. ConCor-1 consistently leads baselines, while architectural studies identify design choices that improve correspondence quality and cross-modal alignment.

  • Results overview: ConCor-1 consistently achieves the best F1-based metrics across image-caption and image-category benchmarks.The benchmarks include COCONut-PanCap, GroundedRef, Flickr30k, COCO, LVIS-minival, and EntitySeg.
  • Findings: ConCor-1 improves JointF1 from 59.9 to 88.8 and mJS from 63.3 to 89.5 on COCONut-PanCap over Qwen3.5-FT.This supports explicit correspondence prediction over autoregressive language-only generation.
  • Scoring function: Bilinear scoring improves thresholded F1 metrics, whereas dot-product scoring performs slightly better on IoU-based metrics.The results suggest bilinear compatibility separates foreground from background tokens, while dot products produce smoother or better-calibrated continuous masks.
  • Attention pattern: Bidirectional attention consistently improves performance across metrics by providing full image-text-bridge context for jointly resolving correspondences.Replacing causal attention benefits correspondence prediction rather than autoregressive token generation.
  • Bridge token assignment: Multi-scale spatial bridge-token assignment performs best overall, while raster-order assignment performs substantially worse.Spatial grid-based assignment yields clear gains, indicating that localized priors help learn stable correspondence queries across object sizes and locations.
  • Qualitative comparison: ConCor-1 better disambiguates referred instances and produces more complete text-image alignments than GDINO+SAM in qualitative examples.GDINO+SAM often detects plausible regions but struggles when multiple people or similar objects appear.

5 Discussion and Future Work

The paper frames grounding as bidirectional concept correspondence, unifying text segments, image segments, and their alignments while positioning ConCor-1 as an initial model and motivating extension to video.

  • Discussion: Bidirectional concept correspondence jointly structures text segments, image segments, and their alignments from an image–text pair.This formulation treats grounding as a single structured output over both modalities and their correspondence.
  • Discussion: The formulation unifies caption grounding, referring-expression grounding, and open-vocabulary grounding under one output space.
  • Discussion: ConCor-1 is a first step that explicitly predicts text masks, image masks, and their correspondences.
  • Future Work: Future work includes extending correspondence to video.

A Limitations and Societal Impact … B.3 Prediction Heads

The paper identifies limitations in mask quality and correspondence-style supervision, while noting both interpretability benefits and risks from grounding errors. ConCor-1 uses learnable bridge tokens and specialized prediction heads to recover correspondence presence, text spans, and image masks from multimodal representations.

  • A Limitations and Societal Impact: Mask quality trails specialized detection and segmentation models because ConCor-1 uses a lightweight image-mask head for joint correspondence prediction.The approach also depends on jointly annotated grounded text spans and image masks, which most existing datasets do not natively provide.
  • A Limitations and Societal Impact: More precise image-text correspondence may improve spatial interpretability in visual search, dataset annotation, assistive perception, and embodied-agent applications.Grounding errors can propagate into downstream decisions, particularly in human-facing or safety-critical settings.
  • B Implementation Details: ConCor-1 follows a grounding-only design, appending learnable bridge tokens and directly predicting image masks, text masks, and correspondence presence.It does not autoregressively decode an output string.
  • B.1 Backbone and Input Sequence: The model uses Qwen3.5-0.8B as a feature-extraction backbone, with bidirectional attention enabling bridge tokens to access complete multimodal context.The backbone is not used as a conditional language generator.
  • B.1 Backbone and Input Sequence: Each image-text pair forms a flat multimodal sequence containing visual tokens, text tokens, and Q bridge tokens, where Nv, Nt, and Q denote their respective counts.Main-table experiments fix the image pixel budget at 1,003,520 while preserving the original aspect ratio through dynamic resizing.
  • B.2 Bridge Tokens and Multi-scale Assignment: Bridge tokens represent candidate correspondences and are organized across grid levels from 1 × 1 to 10 × 10, supporting coarse-to-fine instance representation.Coarse levels can represent large instances, while fine levels can represent smaller localized instances.
  • B.2 Bridge Tokens and Multi-scale Assignment: During training, Hungarian matching assigns each ground-truth correspondence to a bridge token using mask-cell overlap and centroid-to-cell-center distance.The overlap is the primary criterion and centroid distance breaks ties.
  • B.3 Prediction Heads: Three prediction heads share bridge-token features: presence classification, token-level text segmentation, and fine-resolution vision segmentation.The text head uses learned bilinear compatibility, while the vision head decodes dense visual features and scores bridge-to-feature correspondence.

B.4 Training Loss · B.5 Backbone and Input Sequence

ConCor-1 trains bridge-token presence and text/visual segmentation through a four-term weighted objective, while adding modest model and runtime overhead over its backbone. The selected loss weights and computational profile are specified across the two subsections.

  • B.4 Training Loss: The training objective combines presence loss, text segmentation loss, and visual segmentation loss, with the visual term decomposed into sampled-point BCE and Dice components.Together, these form a weighted sum of four terms.
  • B.4 Training Loss: Presence supervision uses binary cross-entropy on every bridge token, labeling matched tokens positive and unmatched tokens negative.The presence term identifies candidate bridge tokens corresponding to valid grounded concepts.
  • B.4 Training Loss: Text supervision applies binary cross-entropy to valid text-token positions using masks derived from character-span overlap, while visual supervision applies only to matched positive bridges.Visual masks use uncertainty-based point sampling and combine point-wise BCE with Dice supervision; unmatched bridges receive only presence loss.
  • B.4 Training Loss: (1, 2, 0.5, 2) is best on all six metrics in the loss-weight sweep and is adopted for the main experiments.The corresponding weights are λtext = 1.0, λbce = 2.0, λdice = 0.5, and λpres = 2.0.
  • B.5 Backbone and Input Sequence: 13.80M additional parameters increase model size by 1.62%, while 385 bridge tokens raise sequence length from 770 to 1,155 and FLOPs by 46.0%.The profile uses a 512×512 image, 512 text tokens, batch size 1, bf16 precision, and an A100-80GB GPU.
  • B.5 Backbone and Input Sequence: 439.9 GFLOPs come from processing bridge tokens through the 24-layer language backbone, while the three prediction heads add 47.8 GFLOPs.The visual segmentation head accounts for most of the prediction-head computation.

B.6 Data Augmentation and Data Mixture · B.7 Inference · B.8 Ablation Details

The paper augments images with collage and crop-based sampling, specifies thresholding and suppression procedures for inference, and standardizes ablation settings across attention, vision, and bridge-token studies. Training-mixture composition and augmentation fractions are reported in Table 8, while ablations use a modified mixture and fixed optimization settings.

  • B.6 Data Augmentation and Data Mixture: Collage augmentation combines multiple images on one canvas, transforms and re-encodes instance masks, and rebuilds text supervision from categories present in the augmented image.It increases grounded instances per sample and broadens object-layout and scale coverage.
  • B.6 Data Augmentation and Data Mixture: The training setup reports source-level mixture weights and the fraction of each source group sampled with collage or crop augmentation in Table 8.The remaining fraction is sampled without image augmentation.
  • B.7 Inference: Inference retains bridge tokens with presence scores above 0.1 as candidate correspondences.The retained set is subsequently processed by the text and visual segmentation heads.
  • B.7 Inference: A text threshold of 0.45 converts sigmoid-activated token-level logits into binary text-token masks.The text segmentation head predicts logits over the input text for each retained bridge token.
  • B.7 Inference: A visual threshold of 0.45 converts bilinearly upsampled sigmoid probabilities into final binary image masks.Visual logits are predicted on a fine 4 × 4 pixel-block grid before upsampling to the original resolution.
  • B.7 Inference: Non-maximum suppression ranks retained predictions by presence score and removes lower-scoring duplicates at an IoU threshold of 0.5.The remaining predictions form the final text–image correspondence set.
  • B.8 Ablation Details: Ablations freeze the vision encoder initially, compare causal with bidirectional attention, then jointly fine-tune vision features for later vision and bridge-token-assignment studies.They use bridge levels 1 × 1 to 5 × 5 and 8 × 8, exclude EntitySeg, Roboflow-VL-100, and GoldG, add LVIS, train for 15K steps, and use NMS IoU 0.9.

C Evaluation metrics details … C.4 Joint Correspondence

The evaluation framework separately measures text segmentation, image segmentation, and joint correspondence using similarity-based matching. Joint correspondence requires the same prediction to achieve both correct text grouping and correct mask segmentation.

  • C Evaluation metrics details: The framework evaluates text segmentation, image segmentation, and joint correspondence through pairwise similarities followed by optimal assignment.The three axes assess span grouping, mask alignment, and cross-modal association, respectively.
  • C.1 Notation: Each ground-truth or predicted group contains character-span sets and a binary image mask, with M ground-truth groups and K predictions.Groups are denoted G_i and Ĝ_j, with span sets T_i and masks S_i.
  • C.2 Text Segmentation: Text segmentation measures whether character spans corresponding to each visual entity are correctly identified, independent of predicted masks.Span IoU is computed after unioning all intervals within each group.
  • C.2 Text Segmentation: Hungarian matching assigns predicted and ground-truth text groups one-to-one using span IoU, retaining all assignments without a similarity threshold.Hard metrics then count matched pairs meeting threshold τ, while the soft metric averages each ground-truth group’s best-matched span IoU.
  • C.3 Image Segmentation: Image segmentation evaluates mask quality independently of character spans by replacing span IoU with pixel-level mask IoU.A separate Hungarian assignment σ is computed independently from text matching.
  • C.4 Joint Correspondence: Joint correspondence evaluates whether the correct text spans are associated with the correct masks in the same prediction.Unlike the independent modality evaluations, joint evaluation requires both text grouping and segmentation to be correct.
  • C.4 Joint Correspondence: The joint score uses the geometric mean of text and mask similarities, sharply penalizing imbalanced predictions.For spanIoU = 0.9 and MaskIoU = 0.1, the joint score is 0.3.
  • C.4 Joint Correspondence: Hungarian matching on the joint matrix produces assignment ρ, which may differ from text-only π and mask-only σ assignments.A joint true positive requires both individual scores to exceed the threshold, making the criterion strict.

D Dataset curation · D.1 Caption data · D.2 Instance segmentation data

The paper curates a unified training mixture combining caption-grounding and instance-segmentation data from human-annotated, converted, and automatically constructed sources. Caption datasets are standardized through rewriting and mask processing, while instance data expands mask supervision to long-tail, dense, and point-annotated scenes.

  • D Dataset curation: The unified training mixture combines human-annotated benchmarks, converted datasets, and automatically constructed supervision for caption grounding and instance segmentation.Caption sources include GoldG, COCONut-PanCap, and GroundedRef; instance sources include COCO, ADE20K, COCONut, EntitySeg, and converted Roboflow-VL-100 data.
  • D.1 Caption data: GoldG and COCONut-PanCap are converted into a consistent caption-grounding format using a shared two-stage LLM pipeline for coreference cleanup and caption rewriting.The pipeline first validates and cleans coreference groups, then minimally rewrites captions for consistent grounding.
  • D.1 Caption data: GoldG combines Flickr30k and GQA annotations, grouping entities into coreference sets and converting box supervision into pseudo instance masks with SAM3.GoldG is sourced from the Grounding DINO training corpus and uses SAM3-generated pseudo masks because its original supervision is box-level.
  • D.1 Caption data: COCONut-PanCap uses relabeled COCO 2017 validation images with panoptic captions and directly decodes masks from RGB-encoded panoptic annotations.Unlike GoldG, it already provides panoptic segment annotations.
  • D.1 Caption data: GroundedRef synthetically generates short referring expressions and compositional queries from COCO 2017 panoptic segmentation to ground captions to instances.The referring expressions disambiguate a target among same-category distractors, while compositional queries combine grounded sub-queries.
  • D.2 Instance segmentation data: 81 instance-oriented datasets from Roboflow-VL-100 are converted into a unified instance-segmentation format by prompting SAM3 with bounding boxes alongside standard segmentation benchmarks.The collection includes COCO, ADE20K, COCONut, and EntitySeg, but the paper notes that standard datasets do not fully cover long-tail categories and dense scenes.
  • D.2 Instance segmentation data: PixMo Points is filtered to retain physical objects with clear spatial extent, after which verbose labels are normalized for instance-segmentation training.The original annotations also contain abstract or non-segmentable concepts such as shadows and reflections.
  • D.2 Instance segmentation data: SA-1B data focuses on high-density images containing 200–500 segments and combines GPT-4o-mini object discovery, MolmoPoint localization, and SAM3 mask generation under point-mask consistency constraints.The construction uses the Tars 0–99 subset and two independent pipelines to identify and localize category instances before generating masks.

E Baseline implementation details

The paper implements baselines spanning detector–segmenter pipelines, unified multimodal models, native grounded segmentation, proprietary MLLMs, and autoregressive correspondence generation. These implementations recover text spans and image masks through model-specific matching, prompting, or serialization procedures.

  • Grounding DINO + SAM: Grounding DINO detects caption-grounded boxes, which are passed to SAM3 for instance segmentation using box prompts by default.Variants also prompt SAM3 with recovered text or jointly with box and text, but box-only prompting is the default because it is more stable.
  • Florence-2: Florence-2 jointly performs localization and segmentation, with predicted phrases matched back to caption character offsets and polygons converted into binary masks.Exact substring matching is attempted first, followed by normalized token-level matching that tolerates minor punctuation and whitespace differences.
  • GLaMM: GLaMM generates grounded descriptions with <p>phrase</p>[SEG] markers and one segmentation mask per [SEG] token, conditioned here on the target caption.The adaptation prompts GLaMM to segment objects mentioned in the caption and matches generated phrases back to character spans.
  • GPT-5.4 + SAM: GPT-5.4 predicts grounded character spans or category names together with bounding boxes, which are converted to masks with SAM3.For image-caption inputs it returns grounded caption spans; for image-category inputs it returns category names mapped to category-list spans.
  • Qwen3.5-FT: Qwen3.5-FT autoregressively generates serialized correspondences containing grounded character spans and polygon representations of corresponding image masks.The baseline tests language-generation supervision without bridge-token hypotheses or dedicated text and image segmentation heads; its text scores are stronger than its mask and joint metrics.

F Effects of Annotation Standardization and Model Architecture

This section separates the effects of annotation standardization from model architecture under controlled fine-tuning conditions. Standardized annotations improve JointF1 for both models, while ConCor-1 retains an architectural advantage that is complementary to standardization gains.

  • Experimental setup: Both ConCor-1 and MM-GDINO were fine-tuned for 15K steps with either original or standardized annotations while other training and evaluation settings remained unchanged.ConCor-1 used its native dynamic-resolution input.
  • Effect of annotation standardization: 9.4 points and 9.8 points: standardized annotations improve image-caption JointF1 for MM-GDINO-FT and ConCor-1, respectively.The standardization recovers omitted grounded entities, consolidates coreferential mentions, and aligns grounded text spans with corresponding instance masks.
  • Effect of model architecture: 3.8 points and 11.6 points: with original annotations, ConCor-1 outperforms MM-GDINO-FT on image-caption and image-category benchmarks, respectively.With standardized annotations, the corresponding gains are 4.2 and 11.6 points.
  • Effect of model architecture: 4.2 points and 11.6 points: with standardized annotations, ConCor-1's gains over MM-GDINO-FT persist on image-caption and image-category benchmarks, respectively.The improvements under both annotation settings indicate that architectural advantages complement annotation-standardization gains.

G Visualizations

The visualizations show that ConCor-1’s bridge tokens specialize across spatial scales and object-level hypotheses, while producing more localized and semantically meaningful cross-modal attention than comparison models.

  • Additional qualitative comparisons: GDINO+SAM often detects plausible regions but fails to associate them with the correct caption mentions, especially for repeated categories or contextual descriptions.These failure cases occur in diverse scenes containing multiple objects and complex captions.
  • Additional text-to-image attention visualizations: ConCor-1 produces more compact and semantically meaningful attention maps than original or fine-tuned Qwen3.5.Original Qwen3.5 attends broadly to backgrounds or distractors, while fine-tuning improves localization but remains noisy in many cases.
  • Additional bridge-token attention visualizations: Different bridge tokens specialize in distinct object-level hypotheses by focusing on localized image regions and corresponding text mentions.The visualizations combine bridge-to-image and bridge-to-text attention with spatial priors and final predictions.
  • Bridge token spatial specialization: Coarse bridge-token groups activate broadly, whereas finer groups become progressively localized to their assigned spatial cells.The groups range from a global 1 × 1 partition to increasingly fine image grids.
Loading 2608.07886v1…