PaperlayerView exact revision on arXiv

Source-linked AI summary

ObjEmbed: Towards Universal Multimodal Object Embeddings

Shenghao Fu, Yukun Su, Fengyun Rao, Jing Lyu, Xiaohua Xie, Wei-Shi Zheng

arXiv:2602.01753v3cs.CV

TL;DR

Fine-grained alignment between image regions and textual phrases remains challenging for multimodal embedding models. ObjEmbed represents each object with semantic and localization-quality embeddings, achieving strong performance across 18 benchmarks and about 20-point gains in local image retrieval.

  • Problem

    Recent embedding models remain limited in encoding objects, aligning them with text, and jointly providing precise localization and semantic discrimination.

  • Method

    ObjEmbed encodes all proposed objects and the full image in parallel, using separate object and IoU embeddings to combine semantic matching with localization quality.

  • Results

    Outperforming global image embedding models by around 20 points on four local-retrieval benchmarks, ObjEmbed also performs strongly across 18 diverse vision benchmarks.

  • Takeaways & Limitations

    ObjEmbed provides a unified object-centric embedding framework for visual grounding, local image retrieval, global image retrieval, and related vision tasks.

  • Takeaways & Limitations

    Because ObjEmbed cannot encode missing objects, its performance depends on proposal recall; access to ground-truth regions yields substantial gains.

Abstract

from arXiv · show

Aligning objects with corresponding textual descriptions is a fundamental challenge and a realistic requirement in vision-language understanding. While recent multimodal embedding models excel at global image-text alignment, they often struggle with fine-grained alignment between image regions and specific phrases. In this work, we present ObjEmbed, a novel MLLM embedding model that decomposes the input image into multiple regional embeddings, each corresponding to an individual object, along with global embeddings. It supports a wide range of visual understanding tasks like visual grounding, local image retrieval, and global image retrieval. ObjEmbed enjoys three key properties: (1) Object-Oriented Representation: It captures both semantic and spatial aspects of objects by generating two complementary embeddings for each region: an object embedding for semantic matching and an IoU embedding that predicts localization quality. The final object matching score combines semantic similarity with the predicted IoU, enabling more accurate retrieval. (2) Versatility: It seamlessly handles both region-level and image-level tasks. (3) Efficient Encoding: All objects in an image, along with the full image, are encoded in a single forward pass for high efficiency. Superior performance on 18 diverse benchmarks demonstrates its strong semantic discrimination.

1. Introduction

ObjEmbed addresses the challenge of aligning image objects with text by producing object-centric embeddings that combine semantic content with localization awareness. Its unified, efficient design supports object-level and global retrieval tasks, with strong results across vision benchmarks after training on 1.3M samples.

  • Object-Centric Representation: The final object matching score combines semantic similarity between object and text embeddings with predicted localization quality.This design targets precise localization and strong semantic discrimination for object retrieval and grounding.
  • Object-Centric Representation: ObjEmbed encodes all image objects as embeddings using RoIs and two special tokens: an object token for fine-grained semantics and an IoU token for localization quality.The model uses an off-the-shelf proposal generator before encoding regions as token sequences.
  • Unified Applications: ObjEmbed supports object detection, referring expression comprehension, local image retrieval, and global image retrieval within one framework.Global image embeddings remain available for conventional image-text retrieval, while object embeddings enable fine-grained retrieval even for small targets.
  • Results: 53.0% mAP on COCO is achieved for object detection after training on 1.3M samples.The result is described as highly competitive with specialist models.
  • Results: 89.5 average accuracy on RefCOCO/+/g is achieved for referring expression comprehension.This result reflects the model’s ability to align language with visual objects.

2. Related Work

Related work centers on contrastive language-image pre-training for scalable image-text alignment and on open-vocabulary object detection for text-described object localization. Prior detectors align object embeddings with text using CLIP distillation, CLIP integration, or deep fusion layers.

  • Contrastive Language-Image Pre-Training: Contrastive language-image pre-training aligns matched image-text embeddings while separating mismatched pairs, providing an effective and scalable route to transferable image representations.Reported improvements include sigmoid loss, curated data, hard negatives, and multi-task learning.
  • Open-Vocabulary Object Detection: Open-vocabulary object detection targets arbitrary objects specified by text queries and seeks to encode objects into embeddings aligned with text embeddings.The passage frames this as a text-conditioned detection and alignment problem.
  • Open-Vocabulary Object Detection: Prior methods align object and text spaces through CLIP feature distillation, CLIP-based modules or backbones, or deep fusion layers for cross-modal alignment.These approaches are presented as distinct strategies for connecting object embeddings to the text space.

3. Method

ObjEmbed fine-tunes Qwen3-VL-instruct with specialized tokens to jointly represent objects, box quality, global images, and text queries. It trains these representations through region- and image-level contrastive learning plus IoU regression, encoding all regions and the image in one efficient forward pass.

  • Architecture: ObjEmbed introduces five special tokens for semantic object embeddings, IoU-based box-quality embeddings, global image embeddings, and local or global text embeddings.The local text token matches object embeddings, while the global text token matches global image embeddings.
  • Object representation: A universal proposal generator supplies 100 proposals per image, whose RoIAlign features are compressed into sequential object tokens for simultaneous encoding.This sequence structure allows the model to encode all objects together efficiently.
  • Box-quality modeling: A dedicated IoU token predicts each bounding box’s quality, avoiding optimization conflicts caused by jointly learning localization and classification with one token.The IoU token follows its corresponding object token in the structured sequence.
  • Efficiency: Each object consumes only 8 tokens, and encoding a 1000-token full image keeps the total sequence below 2000 tokens for FlashAttention-2 acceleration.The full image and all objects are encoded in a single forward pass without autoregressive token prediction.
  • Training objectives: Training combines region-level contrastive learning, image-level contrastive learning, and IoU regression under a weighted objective.Region supervision uses sigmoid focal loss with proposals positive when IoU(pj, Bi) > 0.5, while global supervision uses short and long captions separately.

4. Experiment

Experiments show that ObjEmbed provides strong performance across grounding and retrieval tasks through object-aware representations, while remaining efficient and robust to proposal-network choices. Ablations validate its IoU-aware token design, object instructions, combined image/object objectives, and separate global supervision.

  • Main results: ObjEmbed combines rich language understanding with well-localized predictions and achieves consistently strong performance across all five benchmarks.Traditional detectors localize precisely but use fixed vocabularies, whereas MLLMs understand complex descriptions but have coarse spatial reasoning.
  • Main results: 81.7 points overall on traditional image-text retrieval benchmarks makes ObjEmbed highly competitive despite its relatively small-scale training set.The evaluation covers long-text, short-text, and multilingual retrieval.
  • Ablation studies: 5.2% mAP on COCO is gained by replacing sigmoid-focal classification labels with box IoUs, while decoupled classification and localization tokens address conflicting objectives.The ablation motivates separate object and IoU embeddings for semantic matching and localization-quality assessment.
  • Ablation studies: 2.7 points in local image retrieval are gained by combining image-level and object-level training objectives while maintaining object-detection and REC performance.Separate text tokens improve COCO by +0.3% mAP and RefCOCO by +0.8, while two global image tokens produce the highest global retrieval result of 80.6.
  • Robustness and limitations: ObjEmbed remains robust across 50, 100, and 150 proposals and different proposal architectures, but performance depends on proposal recall because missing objects cannot be encoded.WeDetect-Uni reaches an Average Recall of 66.7 on COCO and 50.8 on LVISv1 val; target-dataset fine-tuning or human-annotated boxes can further improve results.
  • Efficiency: 6.9 fps retrieval and detection inference and 4.0 fps REC inference demonstrate efficiency, compared with 9.5 fps for Qwen3-VL-Embedding-8B and 0.4 fps for Qwen3-VL-4B on REC.ObjEmbed encodes all object and global embeddings in one retrieval pass, uses two passes for REC, and precomputes class text embeddings for detection.

5. Limitation

ObjEmbed’s limitations center on its relatively small training set and the challenge of mining hard negatives without introducing false negatives. Future work could expand pre-training data and develop robust sampling strategies that account for incomplete annotations.

  • Scaling up training data: ObjEmbed is trained on only 1.3M samples because of resource constraints, fewer than those used in CLIP-series models.Broader data collection and larger pre-training datasets could enhance performance.
  • Hard negative mining: Hard-negative mining must improve embedding discrimination while mitigating false negatives caused by annotation incompleteness.Robust hard-negative sampling strategies are identified as a direction for further performance gains.

6. Conclusion

ObjEmbed is an MLLM-based object embedding model built around object-oriented representation, versatile downstream use, and efficient encoding. It represents each object with separate semantic and localization-quality embeddings, reducing learning complexity while preserving encoding efficiency.

  • Model overview: ObjEmbed combines object-oriented representation, versatility, and efficient encoding in an MLLM-based object embedding model.The conclusion identifies these as the model’s defining features.
  • Object representation: Each object uses an object embedding for semantic matching and an IoU embedding for assessing localization quality.The two embeddings provide complementary semantic and localization information.
  • Efficiency and versatility: The decoupled embedding design reduces learning complexity while maintaining encoding efficiency and supports a wide range of downstream tasks.ObjEmbed is described as seamlessly applicable across downstream tasks.

A. Details of Task Instructions

ObjEmbed supports diverse downstream tasks whose object-representation requirements differ. Task-specific instructions guide the model toward context-aware embeddings tailored to each task.

  • Different tasks emphasize distinct object features, from category-level shared semantics in detection to instance-specific distinctions in referring expression comprehension.Referring expression comprehension must distinguish visually similar objects, requiring finer-grained representations.
  • Task-specific instructions are introduced to mitigate conflicts between task requirements and tailor context-aware embedding generation.These instructions guide the model during training.

Object Detection

Object detection identifies every object by recognizing visual characteristics shared across instances of each class. Localization matches each object to its category’s archetypal or defining visual form.

  • Objects are detected by recognizing visual features shared across instances of their respective classes.
  • Object localization matches each object to the archetypal visual form of its category.
  • Detection and identification rely on class-defining visual patterns, attributes, or properties common to each object category.

Referring expression comprehension · Celebrity

The section frames referring expression comprehension as locating the uniquely described object through appearance, position, and relationships with surrounding objects, while Celebrity prompts require identifying recognizable public figures by name.

  • Referring expression comprehension: Referring expression comprehension requires locating the specific object that matches instance-level visual details in the description.These details include distinctive appearance-related properties such as color and texture.
  • Referring expression comprehension: The target object is disambiguated by determining its precise location or placement within the image.Spatial position is treated as a separate cue alongside visual characteristics.
  • Referring expression comprehension: Grounding the reference also depends on the object’s spatial arrangement, context, and interactions with nearby objects.Relational information helps identify the correct instance among surrounding items.
  • Celebrity: Celebrity tasks ask the system to identify the famous or well-known person depicted in an image.The expected response is the public figure’s name.
  • Celebrity: When multiple recognizable celebrities appear, the task requires identifying all of them.The prompt specifically extends celebrity recognition from one individual to every recognizable celebrity in the image.

B. Details of Dataset Annotation

ObjEmbed’s training annotations pair each image with long and short captions plus region descriptions linked to objects of interest. The annotation protocol emphasizes diverse, distinctive, objective descriptions at both region and image levels, with explicit quality handling and formatting constraints.

  • Annotation Structure: Each training image receives a long caption, a short caption, and several regions of interest paired with object descriptions.Captions are designed to be diverse and distinctive to reduce false-negative conflicts during contrastive learning, while excluding subjective or interpretive content.
  • Region-Level Captions: Region descriptions focus on inherent and instance-specific visual details that uniquely identify each object without stating coordinates.Descriptions should be concise, clear, diverse, natural, and centered on the described object; poor-quality instances are labeled “Instance quality is poor.”
  • Image-Level Captions: Short image captions identify primary entities, objects, and surroundings in one natural sentence or phrase under 30 words.The short-caption protocol requires concise, clear, natural descriptions.
  • Image-Level Captions: Long image captions include visual attributes and spatial or object relationships while remaining objective and between 50 and 100 words.They must describe only visible content, avoiding subjective language, atmosphere, or interpretations of intent, actions, or object purpose.

C. Details of Local Image Retrieval Benchmarks

This section defines local image retrieval as matching a query to a small region or object rather than an entire scene, and describes three evaluation benchmarks. The benchmarks vary in query modality, gallery construction, annotation protocol, and retrieval metric.

  • Task definition: Local image retrieval evaluates whether textual or visual queries retrieve images containing a specific object or small region rather than the whole scene.The section evaluates this task on three established benchmarks.
  • SORCE-1K: SORCE-1K contains 1,023 images with textual queries describing small objects, typically under 10% of image area, and uses Recall@1.Each query has exactly one positive image, while complex backgrounds and limited context challenge global image embeddings.
  • REIRCOCO: REIRCOCO contains 4,994 COCO images with object-focused referring expressions and adapts evaluation to text-to-image retrieval using Recall@1.The adaptation omits object localization because global embedding models do not support it.
  • ILIAS: ILIAS supports text-based and image-based local retrieval, using a gallery of 4,715 positive images for 1,232 queries that may match multiple positives.The gallery replaces full evaluation against 5 million distractors, which is computationally infeasible.

D. Visualization

The visualizations show ObjEmbed’s versatility across referring expression comprehension and local image retrieval, including OCR, commonsense reasoning, image-image matching, accurate retrieval, and object localization. Self-annotated examples further illustrate accurate and distinctive generated captions.

  • Referring Expression Comprehension: ObjEmbed demonstrates strong OCR, commonsense reasoning, and image-image matching capabilities in referring expression comprehension visualizations.These results complement its strong performance on standard referring benchmarks, including RefCOCO, RefCOCO+, and RefCOCOg.
  • Local Image Retrieval: In three SORCE-1K queries, ObjEmbed ranks the correct target images first and accurately localizes the queried objects.The visualizations show top-3 retrieved images for each query.
  • Local Image Retrieval: Qwen-VL-Embedding-8B fails to retrieve the correct images in these challenging SORCE-1K cases, whereas ObjEmbed succeeds.This comparison highlights ObjEmbed’s effectiveness on challenging local image retrieval examples.
  • Self-Annotated Data: Self-annotated examples contain captions that are accurate and highly distinctive.The dataset examples use carefully designed prompts and frontier MLLMs, with image-level and object-level captions drawn from SA-1B.
Loading 2602.01753v3…