PaperlayerView exact revision on arXiv

Source-linked AI summary

ExpAlign: Expectation-Guided Vision-Language Alignment for Open-Vocabulary Grounding

Junyi Hu, Tian Bai, Fengyi Wu, Wenyan Li, Zhenming Peng, Yi Zhang

arXiv:2601.22666v1cs.CV

TL;DR

Open-vocabulary grounding needs fine-grained vision-language alignment under weak supervision, beyond what global embeddings provide without costly supervision or cross-attention. ExpAlign uses expectation-based soft token-region pooling and multi-scale consistency regularization, achieving strong detection and segmentation results, including 36.2 APr on LVIS rare categories. Its referring-expression performance remains below specialized grounding models, reflecting a limitation in CLIP-based spatial reasoning.

  • Problem

    Global prompt embeddings lose fine-grained structure, while token-level grounding remains difficult under weak supervision and optimization instability.

  • Method

    ExpAlign uses an Expectation Alignment Head for implicit token and instance selection, plus multi-scale semantic and geometry-aware consistency objectives.

  • Results

    ExpAlign shows strong open-vocabulary detection and segmentation performance across LVIS, ODinW, and RefCOCO/+/g, including 36.2 APr on LVIS rare categories.

  • Takeaways & Limitations

    Expectation-guided soft alignment improves fine-grained correspondence and generalization on long-tail categories without requiring instance-level annotations.

  • Takeaways & Limitations

    On RefCOCO/+/g, ExpAlign trails Grounding DINO-T, likely because CLIP’s text encoder is weaker at positional and relational language.

Abstract

from arXiv · show

Open-vocabulary grounding requires accurate vision-language alignment under weak supervision, yet existing methods either rely on global sentence embeddings that lack fine-grained expressiveness or introduce token-level alignment with explicit supervision or heavy cross-attention designs. We propose ExpAlign, a theoretically grounded vision-language alignment framework built on a principled multiple instance learning formulation. ExpAlign introduces an Expectation Alignment Head that performs attention-based soft MIL pooling over token-region similarities, enabling implicit token and instance selection without additional annotations. To further stabilize alignment learning, we develop an energy-based multi-scale consistency regularization scheme, including a Top-K multi-positive contrastive objective and a Geometry-Aware Consistency Objective derived from a Lagrangian-constrained free-energy minimization. Extensive experiments show that ExpAlign consistently improves open-vocabulary detection and zero-shot instance segmentation, particularly on long-tail categories. Most notably, it achieves 36.2 AP$_r$ on the LVIS minival split, outperforming other state-of-the-art methods at comparable model scale, while remaining lightweight and inference-efficient.

1. Introduction

ExpAlign addresses fine-grained open-vocabulary grounding by replacing inadequate global prompt representations with weakly supervised token-level alignment and expectation-based soft pooling. It combines this alignment with consistency objectives and reports strong detection and segmentation results, including gains on rare categories.

  • Fine-grained localization remains difficult for open-vocabulary detection and segmentation, especially with negation, relations, and compositional descriptions.
  • Global CLIP-style prompt embeddings cannot simultaneously encode attribute binding, spatial relations, and negation under cosine similarity.
  • ExpAlign’s Expectation Alignment Head aggregates token-wise similarities through soft expectation, implicitly selecting informative tokens without instance-level annotations.
  • Multi-positive InfoNCE separates prompt-level semantics, while GACO emphasizes relatively consistent regions within ground-truth masks.
  • 36.2 APr is achieved on LVIS rare categories with EAH, with strong results across LVIS, ODinW, and RefCOCO/+/g under similar pre-training scale and model capacity.On referring-expression benchmarks, ExpAlign outperforms YOLOE but trails Grounding DINO-T.

2. Related Work

Related work progresses from global sentence-level alignment toward token-, phrase-, region-, and geometry-aware objectives. ExpAlign combines soft token-level aggregation with regularization while avoiding heavy cross-attention and explicit token annotations.

  • Sentence-level vision-language alignment supports zero-shot transfer but can lose internal prompt structure needed for fine-grained localization.
  • Token- and phrase-level methods model region–word correspondences to capture finer-grained visual-language semantics.
  • InfoNCE-style and region-level contrastive objectives separate positive and negative multimodal pairs while improving localization quality.
  • Existing geometry-aware regularizers commonly use absolute geometric cues, whereas ExpAlign uses relative instance statistics for coherent alignment without rigid spatial targets.
  • ExpAlign combines soft token-level aggregation with principled regularization, providing supervision-efficient alignment without heavy cross-attention or explicit token annotations.

3. Method

ExpAlign constructs prompt-conditioned spatial alignment maps by expectation-based token aggregation across feature scales, then regularizes their semantic and geometric structure. The resulting training design preserves fine-grained geometry, supports implicit instance selection, and uses auxiliary consistency losses only during training.

  • ExpAlign computes prompt-conditioned alignment maps from flexible language prompts to support region-level detection or segmentation under weak supervision.
  • The framework combines an Expectation Alignment Head, cross-scale consistency regularization, and semantic and geometric auxiliary objectives.
  • 3.2. Expectation Alignment Head: The Expectation Alignment Head converts token-wise similarities into EAMs by assigning higher weights to globally informative tokens and suppressing irrelevant ones.
  • 3.3. Consistency Regularization Module: EAMs are unified across scales through resolution-aligned aggregation, combining semantic evidence while preserving fine-grained geometry.
  • 3.3. Consistency Regularization Module: Multi-positive InfoNCE pools top-1% responses to form prompt-level logits, while GACO shapes relative spatial consistency within ground-truth regions.
  • 3.4. Full Training Objective: Consistency losses complement task-specific detection or segmentation losses during training and are discarded at inference.
  • EAMs have a multiple-instance-learning interpretation in which spatial locations are instances and prompts form bags, yielding permutation-invariant soft pooling.

4. Experiment

ExpAlign is evaluated across zero-shot detection, instance segmentation, downstream fine-tuning, referring-expression comprehension, and ablations of alignment, regularization, and backbone choices. It performs strongly on long-tail detection and segmentation, while remaining weaker on relational referring expressions.

  • Implementation: ExpAlign uses a lightweight multi-scale architecture with frozen DINOv3 and CLIP encoders, token-level text representations, EAH alignment maps, and parameter-free consistency regularization.The model combines YOLOv8-style feature enhancement and standard detection or segmentation heads with token-level alignment modules.
  • Zero-shot detection: 37.1 AP and 36.2 APr are achieved on LVIS minival, with leading rare-category performance using Objects365, GoldG, and RefCOCO referring-expression data.On full LVIS val, performance reaches 29.5 AP and 24.8 APr.
  • Zero-shot detection: 47.7 AP on ODinW13 and 22.4 AP on ODinW35 demonstrate competitive transfer across diverse real-world datasets under comparable model scale.The model uses 60M total parameters, including 26M frozen parameters.
  • Zero-shot segmentation: 29.9 APm overall and 29.0 APm_r on LVIS val are obtained zero-shot without LVIS training images, surpassing reported YOLO-Worldv2-L and YOLOE results.The cited comparison reports 19.8 APm for YOLO-Worldv2-L and 22.6–23.5 APm for YOLOE variants.
  • Downstream transfer: ExpAlign outperforms YOLOE baselines across most COCO metrics under both linear probing and full tuning, indicating effective downstream adaptation.Linear probing freezes the backbone for 10 epochs, whereas full tuning trains all parameters for 80 epochs.
  • Ablation study: EAH improves LVIS performance by 5.2 AP over mean pooling and 2.7 AP over global EOT pooling, while reaching 36.2 APr on rare categories.The largest rare-category gain over mean pooling is 8.9 APr points.
  • Ablation study: Adding geometry-aware consistency yields the best reported ablation result of 37.8 AP at λsem = 0.5 and λgeo = 0.5, whereas excessive semantic weighting can hurt rare-category performance.The results support balancing semantic and geometry-aware regularization.
  • Ablation study: Freezing DINOv3 produces 37.2 AP and 35.9 APr, outperforming YOLOv8 by 1.6 AP, while unfrozen DINOv3 training collapses.The comparison uses identical detection and segmentation heads.

5. Visualization

ExpAlign visualizations show how token-wise spatial maps are weighted and aggregated into prompt-conditioned alignment maps, supporting fine-grained grounding and zero-shot generalization.

  • Token-wise spatial alignment maps are aggregated with learned importance weights to form a prompt-conditioned expectation alignment map.The EAH preserves token-level representations and combines their maps through a soft expectation mechanism.
  • The EOT token remains the dominant alignment signal while informative non-EOT tokens add fine-grained semantic cues that refine spatial structure.Examples include tokens such as knee, high, and socks.
  • Qualitative detection examples cover prompts absent from training data, demonstrating zero-shot generalization across diverse objects and phrases.The examples include household objects, tools, signs, and novel categories.
  • Additional examples show non-trivial referring expression comprehension for complex expressions involving relationships and attributes.The visualizations include expressions such as a monkey on the back of a capybara and a person wearing a helmet.

6. Conclusion

ExpAlign addresses weakly supervised open-vocabulary grounding with expectation-guided token-region alignment and consistency regularization. The framework improves detection and segmentation performance, especially for long-tail categories and zero-shot segmentation, while remaining lightweight and pipeline-compatible.

  • ExpAlign uses an Expectation Alignment Head to perform implicit token selection and soft region alignment without instance-level annotations.Its expectation mechanism aggregates token-level vision-language similarities under weak and ambiguous supervision.
  • Top-K multi-positive contrastive learning and Geometry-Aware Consistency regularization jointly improve semantic discriminability and spatial coherence during training.The objectives are applied as a multi-scale consistency regularization strategy.
  • ExpAlign consistently improves open-vocabulary detection and instance segmentation, particularly on long-tail categories and zero-shot segmentation quality.The method remains lightweight and compatible with standard detection and segmentation pipelines.

A. Connection to Multiple Instance Learning

ExpAlign admits an exact interpretation as attention-based soft MIL pooling: tokens provide shared posterior weights, while spatial locations act as instances aggregated into a prompt-level response.

  • EAH computes token-wise similarities, spatially averages token responses, and forms a token posterior used to weight each location's token-affinity vector.The same linear functional is applied to every instance.
  • In the MIL interpretation, each prompt defines a bag whose instances are the unordered spatial locations in its feature map.The mapping from spatial features to instance scores is permutation equivariant.
  • The resulting operator is permutation invariant and can interpolate between mean, max, and soft-attention pooling by changing temperature and the Top-K ratio.This matches the expressive family of attention-based MIL operators.
  • ExpAlign performs principled MIL-style soft selection over instances while allowing uncertainty at both token and spatial levels.Its auxiliary losses correspond to bag-level discriminative learning and intra-bag energy shaping.
  • The geometry-aware term yields Gibbs reweighting of prompt-patch probabilities under constrained free-energy minimization without explicit instance labels.

B. Variational Derivation of Gibbs Reweighting in Energy-Based Consistency Regularization

The variational formulation defines a free-energy objective over prompt-patch distributions, combining alignment energy, geometry shaping, and KL regularization. Under the stated assumptions, it has a unique Gibbs-form minimizer that yields the geometry-aware training loss.

  • Setup: The formulation operates on a finite collection of prompt-patch pairs and assumes bounded alignment and instance-wise geometry scores.The temperature satisfies τ > 0, while λ is the geometry weight.
  • Variational objective: The free-energy functional is strictly convex on the probability simplex and therefore admits a unique minimizer.Linearity of the energy term and strict convexity of the KL term establish this result.
  • Gibbs solution: The unique minimizer has an explicit Gibbs exponential-family form, with geometry entering the effective logits through the energy expression.Substituting E(p, i) = −S̃_p(i) produces the equivalent alignment-based form.
  • Training loss: Cross-entropy against an empirical target distribution over prompt-patch pairs is equivalent up to normalization constants to the reported geometry-aware loss.The target weights can be based on instance-local geometry scores.
  • Properties: The KL term prevents collapse toward a point mass by enforcing positive entropy, while τ controls the stability-selectivity trade-off and λ controls geometric shaping strength.
  • Properties: Additive energy shifts leave the Gibbs distribution unchanged, whereas positive energy rescaling can be absorbed into the temperature parameter.

C. Comparative Evaluation of ExpAlign, Grounding DINO, and GLIP on Diverse Real-World Datasets in ODinW

ExpAlign shows competitive performance across diverse ODinW datasets, with especially strong gains on uncommon, underrepresented, and domain-specific scenarios. Its results are evaluated against Grounding DINO and GLIP under comparable model-scale settings.

  • 22.4 average score places ExpAlign slightly above Grounding DINO at 22.3 and clearly above GLIP at 19.6 on ODinW.
  • 45.46 on MountainDewCommercial exceeds Grounding DINO at 25.46 and GLIP at 21.60.
  • 7.83 on MaskWearing and 5.23 on PKLot 640 exceed both baselines in uncommon or underrepresented scenarios.The corresponding baseline scores are 0.25 and 1.10 on MaskWearing, and 0.06 and 0.00 on PKLot 640.
  • ExpAlign achieves leading results in several ODinW-13 cases, including CottontailRabbits, EgoHands generic, pistols, and VehiclesOpenImages.The passage connects these observations with enhanced generalization in diverse open-world evaluation settings.

F. EAM Heatmap Visualizations for Negative Prompts

EAM visualizations show positive prompts concentrating activation on the foreground uniform, while negative prompts diffuse activation across the background. This behavior is presented as evidence of negative-prompt suppression in prompt-guided grounding.

  • Negative prompt activations distribute more uniformly across the background instead of concentrating on the foreground uniform.The visualization compares positive “sailor uniform” and negative “black sailor uniform” prompts.
  • Diffusing negative-prompt attention reduces false positives in irrelevant regions and enhances robustness in prompt-guided tasks.
  • Background-dominant activations indicate effective suppression of unseen negative prompts.

G. Impact of Global Negative Vocabulary

Global negative-vocabulary choices affect rare-category performance more than overall, common-category, or frequent-category performance. The paper identifies vocabulary quality measurement and construction as an open challenge.

  • ±0.8% variation in APr occurs on LVIS minival when the negative prompt set changes.Changes include sampling strategies, vocabulary sizes, and semantic distributions.
  • ±0.2% variation affects overall AP, APc, and APf under the same negative-vocabulary changes.
  • Rare-category representations are more sensitive to negative-prompt interference than frequent and common categories.Semantically close negatives may induce stronger gradient conflicts during contrastive alignment.
  • No standardized metric or principled methodology currently quantifies global negative-vocabulary quality or difficulty in open-vocabulary detection.The paper identifies embedding-aware sampling, online hard-negative mining, and dynamic vocabulary curation as future directions.

H. More Visualization Examples

Additional visualizations demonstrate ExpAlign’s zero-shot grounding across diverse scenes with multiple objects and detailed prompts. The examples include novel categories and complex compositions with instance-mask predictions.

  • ExpAlign demonstrates strong open-vocabulary grounding and precise instance masks across novel categories and complex compositions.
  • Figure 5 presents prompts for clothing, winter-sports, character, and toy-related objects across four visual examples.
  • Figure 6 presents prompts for lanterns, clothing, animals, toys, documents, people, and accessories across four visual examples.
Loading 2601.22666v1…