Source-linked AI summary
ROSE: Retrieval-Oriented Segmentation Enhancement
Song Tang, Guangquan Jie, Henghui Ding, Yu-Gang Jiang
TL;DR
MLLM-based segmentation struggles with novel and emerging entities because its knowledge cannot be updated easily. The paper introduces NEST and its automated benchmark, then proposes ROSE to retrieve current textual and visual information and enhance segmentation. ROSE improves NEST performance, including a 19.2% gIoU advantage over a Gemini-2.0-Flash retrieval baseline.
Problem
MLLM-based segmentation models struggle to segment novel entities absent from training data and emerging entities requiring up-to-date external knowledge.
Method
The paper introduces NEST, constructs a continuously updated benchmark, and proposes ROSE, a plug-and-play framework with four retrieval and prompt-enhancement components.
Results
19.2% gIoU is ROSE’s reported advantage over a strong Gemini-2.0-Flash search baseline on NEST.
Takeaways & Limitations
ROSE improves novel-emerging segmentation while maintaining competitive results on standard benchmarks.
Abstract
from arXiv · showhide
Existing segmentation models based on multimodal large language models (MLLMs), such as LISA, often struggle with novel or emerging entities due to their inability to incorporate up-to-date knowledge. To address this challenge, we introduce the Novel Emerging Segmentation Task (NEST), which focuses on segmenting (i) novel entities that MLLMs fail to recognize due to their absence from training data, and (ii) emerging entities that exist within the model's knowledge but demand up-to-date external information for accurate recognition. To support the study of NEST, we construct a NEST benchmark using an automated pipeline that generates news-related data samples for comprehensive evaluation. Additionally, we propose ROSE: Retrieval-Oriented Segmentation Enhancement, a plug-and-play framework designed to augment any MLLM-based segmentation model. ROSE comprises four key components. First, an Internet Retrieval-Augmented Generation module is introduced to employ user-provided multimodal inputs to retrieve real-time web information. Then, a Textual Prompt Enhancer enriches the model with up-to-date information and rich background knowledge, improving the model's perception ability for emerging entities. Furthermore, a Visual Prompt Enhancer is proposed to compensate for MLLMs' lack of exposure to novel entities by leveraging internet-sourced images. To maintain efficiency, a WebSense module is introduced to intelligently decide when to invoke retrieval mechanisms based on user input. Experimental results demonstrate that ROSE significantly boosts performance on the NEST benchmark, outperforming a strong Gemini-2.0 Flash-based retrieval baseline by 19.2 in gIoU.
1. Introduction
The paper defines NEST to test segmentation of novel and emerging entities that MLLMs cannot recognize from existing or current knowledge. It also introduces an automated benchmark and ROSE, a retrieval-oriented plug-in framework for addressing this gap.
- Motivation: MLLM-based segmentation models struggle with newly emerging information because frequent data collection, cleaning, and retraining are impractical.This limitation causes failures on entities that appear after the models’ training data or knowledge cutoff.
- Novel Emerging Segmentation Task: NEST evaluates binary-mask segmentation for novel entities absent from training data and emerging entities requiring up-to-date knowledge.The task specifically tests retrieval, recognition of unseen entities, and application of retrieved knowledge.
- Benchmark: The NEST benchmark contains over 1,500 image-question-answer-mask pairs generated through a continuously updated web-data pipeline.The automated process supports scalable, timely evaluation as new concepts and objects emerge.
- ROSE: ROSE is a plug-and-play framework with Internet Retrieval-Augmented Generation, Textual Prompt Enhancer, Visual Prompt Enhancer, and WebSense modules.These components retrieve current multimodal information, enrich prompts, provide reference images, and selectively activate retrieval.
- Results: 19.2% gIoU separates ROSE from a Gemini-2.0-Flash search baseline on the NEST benchmark.The reported comparison appears in the stated contribution summary.
2. Related Work
The related-work section situates the paper within referring expression segmentation, tracing the field’s progression from CNN-LSTM feature fusion to transformer-based attention.
- Referring Expression Segmentation: Referring Expression Segmentation segments image targets specified by textual descriptions.Early methods extracted visual features with CNNs and encoded language using LSTMs before fusing the representations.
- Transformer-based methods: VLT reformulated referring segmentation as an attention problem using language features to query visual features.This marked the introduction of transformer architecture into the described RES progression.
- Method evolution: The section presents multimodal representation fusion as a recurring design pattern in earlier referring-segmentation systems.The passage contrasts simple concatenation or related operations with later attention-based processing.
3. NEST Dataset
NEST is built as a continuously refreshed benchmark because fixed datasets risk future data leakage and are costly to annotate comprehensively. Its automated engine converts trending web information into multimodal segmentation samples.
- Dataset motivation: Fixed NEST evaluations can become vulnerable to training-data leakage as future MLLMs incorporate previously collected examples.Continual manual collection and segmentation annotation are also described as labor-intensive.
- Sample structure: Each NEST sample contains a question, textual answer, reference images, and segmentation masks.This multimodal structure supports evaluation of novel and emerging entity segmentation.
- Query generation: The data engine uses Google Trends, manual additions, and LLM filtering to assemble segmentable queries across underrepresented domains.Filtering removes trending concepts that are not directly associated with segmentable objects.
- VQA construction: News articles are temporally and semantically filtered before LLMs generate natural questions whose answers are the original query terms.The resulting VQA triplets combine generated questions, answers, and multi-entity images without explicitly naming the query in the question.
- Auto Labeling: CLIP similarity selects the target proposal, and SAM then generates its segmentation mask when similarity exceeds threshold τ.The automated labeling pipeline compares detected entities in multi-entity images with representations from single-entity reference images.
- Dataset statistics: The collected dataset contains 1,548 evaluation samples spanning people and products across economics, technology, politics, entertainment, sports, and society.Each image contains 2.7 valid entities on average, while each image is paired with 1.6 unique questions on average.
4. Method
ROSE selectively retrieves current textual and visual information, uses it to enrich prompts and identify novel targets, and then produces segmentation masks through MLLM-based and visual modules.
- Textual Prompt Enhancer: TPE combines the user query, retrieved answer, and extra background knowledge into an enhanced textual prompt for segmentation.The resulting prompt is designed to improve the model’s understanding and identification of the target object.
- Internet Retrieval-Augmented Generation: IRAG generates optimized search queries, retrieves web content, embeds chunks in a vector database, and synthesizes candidate answers.It then uses the input image to narrow candidate answers before retrieving relevant reference images.
- Multimodal disambiguation: Google Cloud Vision extracts image entities so retrieved answer candidates can be matched without relying on MLLMs for novel-entity recognition.The method explicitly avoids MLLM entity extraction because the paper states that MLLMs cannot accurately identify novel entities.
- Visual Prompt Enhancer: VPE clusters retrieved images, derives a CLIP prototype, matches it against detected image entities, and sends the selected box to SAM’s mask decoder.A confidence threshold τ determines whether the most similar detected entity is designated as the target.
- WebSense: WebSense decides whether internet retrieval is necessary for a given image-query input.Selective activation is intended to reduce computational and network resources while preserving retrieval for queries needing external knowledge.
5. Experiments
ROSE is evaluated on NEST and mixed NEST+ settings, where retrieval-oriented enhancements improve novel and emerging segmentation while preserving competitive performance on traditional tasks.
- NEST evaluation: More than 30% gIoU improvement is achieved on NEST by retrieving up-to-date information for entities beyond existing models’ knowledge cutoffs.The experiments report that existing works fail on this task, whereas ROSE uses Retrieval-Augmented Generation to identify and segment novel or emerging entities.
- NEST evaluation: ROSE significantly outperforms the two-stage GPT-4o mini Search and Gemini-2.0 Flash Search baselines.The reported explanation emphasizes richer target descriptions and background knowledge from TPE, together with internet-sourced images for novel entities.
- Mixed-dataset evaluation: ROSE significantly enhances the NEST split while maintaining competitive results on ReasonSeg and RefSeg in the mixed NEST+ evaluation.NEST+ combines NEST with ReasonSeg, RefCOCO, RefCOCO+, and RefCOCOg to evaluate retrieval-based and traditional referring segmentation together.
- Ablation study: Adding IRAG increases overall gIoU by 7.0%, while TPE improves emerging-entity gIoU by 6.2% and VPE improves novel-entity cIoU by 24.5%.The ablation also reports a 13.0% overall gIoU gain from VPE and attributes these gains to retrieved information, structured prompts, and visual references.
- Qualitative results: Qualitative examples show ROSE segmenting unseen Xiaomi SU7 and correctly identifying the actual MLB player when LISA produces no output or selects the wrong person.The comparisons involve LISA and READ on both novel entities and emerging entities requiring current knowledge.
6. Conclusion
The paper introduces NEST, an automated real-time benchmark, and ROSE, a retrieval-enhanced framework for MLLM-based segmentation. Experiments show improved novel and emerging segmentation while maintaining competitive performance on standard benchmarks.
- NEST evaluates segmentation of entities absent from MLLM training data and entities requiring up-to-date external knowledge.
- The NEST dataset is built from real-time image-news pairs collected through an automated pipeline.
- ROSE enhances MLLM-based segmentation through real-time internet retrieval in a plug-and-play approach.
- ROSE significantly improves novel emerging segmentation while maintaining competitive results on standard benchmarks.