PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic-Spatial Discriminability Enhancement for Generalized Visual Grounding

Kaiyan Lei, Xu-Yao Zhang

arXiv:2608.30233v1cs.CV

TL;DR

GVG must localize zero, single, or multiple targets despite visually similar instances and limited coarse-grained semantic or spatial cues. SSDE combines fine-grained visual-attribute enhancement with instance-center structural supervision, and the paper reports superior performance across ten classic and generalized visual-grounding datasets. Its spatial formulation assumes that the largest connected component preserves the most complete and salient instance structure.

  • Problem

    Coarse sentence-level semantics and regional interactions can confuse visually similar targets in complex multi-target grounding, hindering stable instance-level decision boundaries.

  • Method

    SSDE combines SeDE's spatially guided integration of fine-grained visual attributes with SpDE's auxiliary supervision of an instance center density map.

  • Results

    SSDE achieves superior performance across ten datasets spanning classic and generalized visual grounding tasks.

  • Takeaways & Limitations

    Jointly modeling fine-grained semantic attributes and structured spatial distributions improves cross-modal understanding and instance-level localization.

  • Takeaways & Limitations

    The spatial supervision assumes the largest connected component preserves the instance's most complete and salient structure.

Abstract

from arXiv · show

Generalized Visual Grounding (GVG) task aims to localize targets in an image based on referring expressions, extends the classical visual grounding paradigm by integrating multi-target and non-target scenarios. Previous methods typically rely on global semantic matching or coarse-grained region interactions for localization, where the discriminative cues are primarily derived from sentence-level semantics or regional context. In complex multi-target scenarios, such approaches tend to confuse visually similar targets, making it difficult to establish stable instance-level decision boundaries. To address these limitations, this paper proposes a novel Semantic-Spatial Discriminability Enhancement (SSDE) framework for generalized visual grounding, which aims to enhance the discriminative ability on fine-grained semantics and spatial localization, improving both cross-modal understanding and instance-level grounding. Specifically, to enhance the semantic discriminability of query representations at the fine-grained level, we propose a Semantic Discriminability Enhancement (SeDE) module, which leverages spatially guided cross-attention to disentangle fine-grained target-relevant visual attributes and integrates them with the textual subject semantics. Furthermore, to strengthen the spatial discriminability of the referred targets, we introduce a Spatial Discriminability Enhancement (SpDE) module, which models an instance center density map to characterize the spatial distribution of targets, and explicitly constructs instance separation structures in the spatial domain by employing them as an auxiliary supervision signal. Extensive experiments show that SSDE achieves superior performance on ten datasets across both classic and generalized visual grounding tasks.

1 Introduction

GVG extends visual grounding to multi-target and non-target cases, where coarse sentence-level semantics and global masks can confuse similar instances. SSDE addresses this by jointly enhancing fine-grained semantic cues and instance-level spatial structure through SeDE and SpDE.

  • GVG extends classic visual grounding from single-target localization to multi-target and non-target cases.
  • Sentence-level semantics struggle to distinguish best-matching targets from partially relevant candidates amid numerous distractor regions.
  • Global masks and count heads provide unstable instance-level boundaries because target-count differences are weakly propagated into spatial localization.
  • Fine-grained visual attributes such as color, shape, texture, and relational descriptors mitigate semantic ambiguity among similar targets.
  • SSDE combines SeDE, which integrates spatially guided visual attributes with textual semantics, and SpDE, which imposes instance-level structural constraints through center responses.
  • Experiments across ten CVG and GVG benchmarks report state-of-the-art performance for SSDE over existing methods.

2 Related Work

Prior visual grounding work progressed from proposal- or anchor-based matching to Transformer cross-modal modeling, while GVG research broadened the task to zero, single, and multiple targets. Recent methods address region dependencies, attention redundancy, and semantic understanding in these generalized settings.

  • Classic visual grounding methods localize a single target using region proposals, dense anchor-based detections, or Transformer cross-modal relationships.
  • GVG includes expressions referring to zero, single, or multiple targets, extending visual grounding beyond the classic single-target setting.
  • Recent GVG methods model long-range region dependencies, guide attention with region-aware anchors, and improve robustness to non-target regions.

3 Methods

SSDE combines a multimodality encoder with SeDE and SpDE to improve fine-grained semantic discrimination and instance-level spatial localization. SeDE enriches query representations with spatially selected visual attributes, while SpDE supervises instance-center structure alongside the main grounding outputs.

  • 3.1 Overview: SSDE uses a multimodality encoder backbone together with SeDE and SpDE for semantic and spatial discriminability enhancement.The framework produces grounding outputs while adding auxiliary supervision for similarity and instance-center maps.
  • 3.2 Semantic discriminability enhancement: Learnable queries aggregate textual features through cross-attention to form text-driven, subject-aware representations.Max pooling over textual features adds global context to token-level semantics.
  • 3.2 Semantic discriminability enhancement: Spatially guided attention uses image-text correspondence to extract fine-grained attributes such as color, texture, and pose from target-related regions.The similarity map is supervised by the ground-truth mask and transformed into an attention bias for semantically relevant regions.
  • 3.2 Semantic discriminability enhancement: The attribute features are transformed and fused with subject semantics to generate queries that support instance-level differentiation.Subject semantics determine the category, while attribute details distinguish individual instances.
  • 3.3 Spatial discriminability enhancement: SpDE predicts an instance-center density map as an auxiliary objective to impose structural constraints and separate multiple instances spatially.Centers are derived from the largest connected components, expanded with Gaussian kernels, and aggregated by point-wise maximum.
  • 3.4 Loss: The density map represents all instance centers and encodes target-count information, while auxiliary losses supervise both similarity and center predictions.This supervision is combined with the main detection, segmentation, and target-presence objectives.

4 Experiments

SSDE is evaluated across classic and generalized visual grounding benchmarks, where its semantic and spatial modules consistently improve localization performance. Ablations further show that fine-grained visual attributes and instance-centered density supervision complement each other.

  • Comparison with SOTA Methods: SSDE consistently outperforms existing methods across REC, RES, GREC, and GRES benchmarks.It reports superior performance on both classic and generalized visual grounding tasks, including comparisons against methods using larger models, additional pretraining data, or independent modality encoders.
  • Generalized visual grounding: 11.0%, 9.0%, and 7.7% average gains over HieA2G are reported across the three GREC splits.Against PropVG with the same backbone, SSDE also achieves average gains of 2.6%, 2.7%, and 2.0%.
  • Ablation studies: Both SeDE and SpDE improve performance, while their combination yields the best results.The ablation attributes the gains to visual-attribute-enhanced query representations and explicit supervision from instance center density maps.
  • Ablation studies: Spatially guided cross-attention with scaling factor m produces the most reliable target-relevant attribute extraction within SeDE.Standard cross-attention is more susceptible to background noise, while adding spatial priors without scale regulation is less effective.
  • Ablation studies: Instance-centered density maps provide clearer multi-instance spatial structure than similarity maps alone.Each instance forms a distinct local peak, whereas similarity maps tend to cover entire target regions and lack clear instance separation.

5 Conclusion

SSDE jointly enhances semantic and spatial discriminability for generalized visual grounding, targeting improved cross-modal understanding and instance-level localization. Its modules use fine-grained visual attributes and instance-center distributions to support separable multi-target grounding, with state-of-the-art results on multiple benchmarks.

  • SSDE jointly enhances semantic and spatial discriminability to improve cross-modal understanding and instance-level localization.
  • SeDE incorporates fine-grained visual attributes through spatially guided cross-modal interaction to enlarge inter-instance discriminative margins.
  • SpDE models instance-center distributions and uses auxiliary supervision to impose structural constraints that promote target separability and independence.
  • SSDE achieves state-of-the-art performance on multiple visual grounding benchmarks.

A Datasets

The appendix describes classic and generalized visual grounding datasets and their evaluation metrics. The datasets vary in expression complexity, target cardinality, and inclusion of negative or non-target cases.

  • RefCOCO and RefCOCO+ contain roughly 142,000 referring expressions across about 50,000 objects and 20,000 images, with testA focused on multiple people.
  • RefCOCOg uses longer, more complex expressions averaging 8.4 words and contains 85,474 expressions for 54,822 objects across 26,711 images.
  • gRefCOCO extends RefCOCO with 80,022 multi-target and 32,202 non-target expressions for generalized grounding evaluation.
  • Ref-ZOM covers one-to-zero, one-to-one, and one-to-many scenarios corresponding to no-target, single-target, and multi-target settings.
  • R-RefCOCO augments training data with negative expressions at a 1:1 ratio relative to positive expressions.
  • CVG uses Precision@0.5 for REC and mIoU for RES, while GREC uses Pr@(F1=1, IoU≥0.5) and N-acc. as main metrics.

B.2 Experiment setup

The experiment setup uses a BEiT-3 multi-modality encoder with multi-scale visual features, alongside a documented REC table protocol and Precision@0.5 evaluation.

  • The model uses a pre-trained BEiT-3 ViT-B multi-modality encoder with patch size 16 and channel dimension 768.
  • Visual features from backbone layers 4, 8, and 12 are fed into an FPN to obtain multi-scale representations.
  • Training uses Adam with learning rates of 5 × 10^-5 for the multi-modality encoder and 5 × 10^-4 for remaining parameters, using 320 × 320 inputs and batch size 48.
  • Table 9 compares REC methods on RefCOCO/+/g using Precision@0.5, with backbone, fine-tuning, and multi-task columns documented.

C.1 Results on REC

SSDE outperforms competing REC methods across the RefCOCO/+/g benchmarks, including larger SAM- and LLM-based systems and similarly scaled state-of-the-art models.

  • Up to 4.9%, 7.5%, and 4.9% average gains across three split sets are reported against task-specific methods HieA2G and OneRef.
  • SSDE outperforms larger SAM-based and LLM-based methods as well as task-specific models trained with additional datasets.

C.2 Ablation Studies

The ablations examine instance-center selection, auxiliary-loss weighting, query count, and efficiency. Results favor largest-component centers, balanced auxiliary weights, 10 queries, and a strong efficiency–performance trade-off.

  • Instance center determination: Selecting the largest connected component provides the most reliable instance center for reference-point initialization under occlusion.Using all components can assign multiple centers to one instance, while using the central component can bias localization toward fragmented regions.
  • Auxiliary-loss weighting: The optimal auxiliary-loss setting is α=0.1 and β=0.1, balancing semantic alignment with instance-center spatial modeling.Lsim enforces cross-modal semantic alignment, whereas Licdm models the spatial distribution of instance centers.
  • Loss-term ablation: The final loss weights are 0.2, 1.0, 0.2, 0.1, and 0.1, achieving the reported optimal trade-off in grounding accuracy.These weights cover DETR-style, instance-segmentation, existence, and auxiliary structural-regularization losses after tuning on gRefCOCO.
  • Query-number ablation: 10 queries produce the best GRES performance, while excessive queries dilute semantic focus and increase optimization difficulty.The paper attributes the advantage of 10 queries to more stable cross-modal coupling within the feature space.
  • Efficiency comparison: SSDE achieves the best overall performance with relatively low parameter count and computational cost compared with other visual grounding methods.Relative to InstanceVG, it adds only marginal parameters and overhead while delivering substantially superior performance.

D Visualization

The visualizations show SSDE localizing referred targets accurately across complex expressions and generalized grounding settings. Its predictions remain effective for multiple targets, missing targets, dense similar instances, and scenes with more than three targets.

  • CVG visualization: SSDE accurately localizes targets for long and complex expressions, combining fine-grained visual attributes with subject-level textual semantics.The qualitative results are presented on RefCOCO/+/g.
  • GVG visualization: SSDE maintains high accuracy across single-target, multi-target, and non-target scenarios, including missing targets and densely distributed similar instances.The gRefCOCO visualizations show detection and segmentation in a unified manner.
  • Multi-instance visualization: On Ref-ZOM, SSDE accurately localizes all relevant instances in complex scenes containing more than three targets.The results indicate robust understanding of referring expressions in multi-instance settings.
  • Qualitative interpretation: Fine-grained visual attributes improve inter-instance semantic discrimination, while instance-level structural constraints preserve spatial independence.Together, these mechanisms mitigate instance adhesion and spatial ambiguity in the qualitative examples.
Loading 2608.30233v1…