PaperlayerView exact revision on arXiv

Source-linked AI summary

High-Quality Entity Segmentation and Grounding

Lu Qi, Yi-Wen Chen, Tao Zhang, Xiangtai Li, Xu Yang, Bo Du, Ming-Hsuan Yang

arXiv:2402.02555v2cs.CVcs.CL

TL;DR

High-quality entity segmentation and semantic grounding face data and integration challenges. ESG combines EntitySeg and a decoupled CropFormer–GELLA pipeline, with reported performance at or above state-of-the-art methods across five tasks. GELLA can use masks from segmentation frameworks beyond CropFormer.

  • Problem

    Entity segmentation is limited by a lack of high-quality data and effective methods, while semantic grounding must be integrated with segmentation efficiently as entity counts grow.

    Source: §1 · p. 1

  • Method

    ESG combines the EntitySeg dataset and CropFormer for entity segmentation with GELLA, which uses a large language model to ground entities from captions.

    Source: §1 · pp. 1–2

  • Results

    Across five tasks, ESG outperforms or is comparable to state-of-the-art methods while retaining user interaction.

    Source: §1 · p. 2

  • Takeaways & Limitations

    GELLA can process masks from segmentation frameworks beyond CropFormer, keeping the grounding module flexible.

    Source: §1 · p. 2

    GELLA may omit ground-truth nouns, captions may not cover all segmented entities, and the current model cannot handle part segmentation or its association.

    Source: §5.5 · p. 15

Abstract

from arXiv · show

In this work, we propose ESG, a pipeline for high-quality entity segmentation and grounding supported by a new dataset EntitySeg. At first, the proposed dataset naming EntitySeg contains images spanning various image domains and entities, along with plentiful high-resolution images and high-quality mask annotations for training and testing. Then, the ESG mainly consists of two modules: CropFormer for high-quality entity segmentation whereas GELLA for accurate noun extraction from sentences and semantic matching between language and visual regions. Unlike existing grounding methods that jointly train a segmentation and a large language model, ESG adopts a two-stage decoupled design, preserving high-quality masks and grounding robustness without the trade-offs often introduced by joint training. CropFormer ensures high-quality entity segmentation results, which can then be encoded into the GELLA model for effective grounding. Extensive experimental results demonstrate the effectiveness of our proposed pipeline across five tasks, including entity segmentation, panoptic segmentation, open-vocabulary segmentation, referring segmentation, and panoptic localized narratives. Furthermore, GELLA module of ESG pipeline is highly flexible and capable of processing mask inputs from any segmentation framework, thanks to its lightweight colormap/vision encoder, language/mask decoder, and association module. The entity segmentation dataset and grounding code will be released at https://github.com/qqlu/Entity.

1 INTRODUCTION

ESG pairs high-quality entity segmentation with language-guided grounding, supported by EntitySeg and a decoupled CropFormer–GELLA design. The introduction reports strong results across five tasks while retaining user interaction.

1 INTRODUCTION

  • ESG addresses limited segmentation quality and the challenge of integrating semantic grounding with segmentation while keeping inference efficient as entity counts grow.The introduction identifies insufficient high-quality segmentation data and effective methods as part of the problem.Source: §1 · p. 1
  • Across five tasks, ESG outperforms or is comparable to state-of-the-art methods while retaining the ability to interact with users.The tasks include referring expression segmentation, panoptic narrative grounding, panoptic segmentation, reasoning segmentation, and open-vocabulary segmentation.Source: §1 · p. 2
  • EntitySeg supplies high-quality masks from varied image domains and resolutions, enabling unified evaluation of segmentation-model generalization and robustness.The dataset emphasizes high-resolution images and high-quality annotations.Source: §1 · p. 2
  • CropFormer addresses the poor fit of conventional training to EntitySeg’s high-resolution images by associating the same entities across crops with shared queries.More than 70% of EntitySeg images are described as very high resolution.Source: §1 · p. 2
  • GELLA grounds semantic nouns from captions to entity masks using visual and text embeddings produced by its encoders and decoders.Its design includes two single-modal encoders and two dual-modal decoders, followed by an association module.Source: §1 · pp. 2–3
  • The decoupled pipeline lets CropFormer’s high-quality segmentation results feed GELLA, which can also process masks from other segmentation frameworks.This separates segmentation from grounding rather than requiring the two capabilities to be jointly trained.Source: §1 · pp. 1–2

2 RELATED WORK

Related work identifies a need for segmentation across diverse domains and unseen objects, alongside a gap in natural-language grounding at fine pixel-level detail. The paper positions its approach as a decoupled route to generalized entity segmentation and grounding.

Image Segmentation Dataset

  • Existing datasets support varied segmentation tasks, but high-quality, diverse images and class-agnostic coverage of unseen objects remain important needs; ADE20K has low-resolution images from narrow domains.EntitySeg collects images from indoor, outdoor, street, cartoon, and remote domains.Source: §2 · p. 3

Scene Parsing Methods

  • Query-based segmentation methods lack exact locations for ensembling augmented views; the approach links entities across views and combines full-image and local-crop results.This is motivated by the difficulty of ensembling multiple results from augmented image views.Source: §2 · pp. 3–4

Open Image Segmentation; Large Multimodal Model

  • Open-vocabulary methods do not associate images with natural-language prompts, while language-model grounding approaches may lack fine-grained segmentation; this work decouples segmentation and grounding for generalized entities.The related-work discussion distinguishes bounding-box generation and joint-training approaches from the paper's two-stage design.Source: §2 · p. 4

3 ENTITYSEG DATASET

EntitySeg is a high-resolution, diverse entity-segmentation dataset with detailed masks and open-vocabulary labels. Its annotation and collection choices produce broad entity coverage, while comparisons report more entities per image and more complex shapes than COCO and ADE20K.

3 ENTITYSEG DATASET

  • EntitySeg contains 33,227 images with high-quality masks; 71.25% have at least 2000×2000 pixels and 86.23% at least 1000×1000 pixels.The dataset treats semantically coherent regions as entities, including blurred or hard-to-recognize regions.Source: §3 · p. 4

3.1 Image Collection

  • EntitySeg draws from public datasets and Internet sources across multiple domains; its resolutions range normally from 0px to 15,000px, while COCO and ADE20K images are all below 1000px.Figure 3 summarizes resolution distributions and EntitySeg image sources.Source: §3.1 · p. 4

3.2 Image Annotation

  • Annotators first create non-overlapping class-agnostic entity masks, then assign labels from an expandable vocabulary, using “unknown” or “blurred” when needed and “supercategory other” for uncertain fine-grained classes.This reverses the predefined-class-first workflow used by COCO and ADE20K.Source: §3.2 · p. 4
  • EntitySeg annotations cover more image area and contain more masks on average than COCO-Panoptic and ADE20K-Panoptic, reflecting annotation of all semantically coherent regions.The authors connect this annotation approach to human vision, which they describe as class-agnostic.Source: §3.2 · p. 4
  • A repeat annotation study on 500 images, or 1.5% of EntitySeg, found high consistency in both class-agnostic mask annotation and category labeling.The study involved two annotators returning four months after the initial annotation round.Source: §3.2 · p. 4

3.3 Dataset Statistics

  • EntitySeg annotates images at the entity level regardless of thing-or-stuff status, and examples include fine-grained boundaries such as a railing mask.The paper illustrates these annotations with low- and high-resolution photos.Source: §3.3 · pp. 4–5
  • 18.1 entities per image on average in EntitySeg exceeds COCO's 11.2 and ADE20K's 13.6, and EntitySeg entities have more complex shapes than those in both datasets.The paper reports these comparisons in Table 1.Source: Table 1 · p. 5, §3.3 · p. 5
  • The EntityClass subset contains 11,580 open-vocabulary-labeled images, organized into 535 thing and 109 stuff classes.The class set is continually expanded and organized according to Word-Tree.Source: §3.3 · p. 5

4 THE ESG PIPELINE

ESG separates class-agnostic entity segmentation from language grounding: CropFormer fuses full-image and crop views, while GELLA associates semantic nouns with visual entities. GELLA combines image, colormap, and language features, and its training data includes image-description and noun-extraction instructions.

4.1 CropFormer for Segmentation

  • CropFormer fuses full-image and corner-crop views, using shared batch queries to represent the same entities consistently across views.At inference, it averages mask predictions from the full image and four corner crops; entity confidence comes from batch-level entityness scores.Source: §4.1 · pp. 5–7

4.2 GELLA for Entity Grounding

  • GELLA encodes image and entity-colormap features, combines them with language tokens, reconstructs masks, and associates noun embeddings with entity embeddings.Its mask decoder retains mask reconstruction, while the language decoder extracts semantic nouns as <SEG> token embeddings for association.Source: §4.2 · pp. 7–9
  • GELLA preserves contextual information for grounding nouns that refer to multiple entities or specify relationships, such as “the three dogs in the grass.”The authors report that ablations on reasoning and open-vocabulary segmentation support the effectiveness of the GELLA framework.Source: §4.2 · p. 7

4.3 Data Preparation

  • Training data uses GPT-4 prompts for image descriptions and semantic-noun extraction, with COCO Panoptic Narrative Grounding data for noun-to-entity associations.For image-description training, only the language part is trained, without segmentation output.Source: Table 3 · p. 8, §4.3 · p. 9

5 EXPERIMENTS

The experiments cover EntitySeg segmentation benchmarks, CropFormer ablations, and GELLA analyses across grounding and segmentation tasks.

5 EXPERIMENTS

  • The experiments benchmark state-of-the-art approaches on EntitySeg across segmentation tasks.The benchmark includes semantic, instance, panoptic, and entity segmentation.Source: §5 · p. 9
  • CropFormer ablations assess its effectiveness for high-quality entity segmentation.The experiments also report results on other datasets.Source: §5 · p. 9
  • GELLA is analyzed on panoptic narrative grounding, referring expression segmentation, and panoptic segmentation.The experiments also include qualitative visualizations of GELLA.Source: §5 · p. 9

5.1 EntitySeg Dataset Benchmarks

The section benchmarks segmentation on EntitySeg and evaluates how EntitySeg pretraining transfers to other segmentation settings.

5.1 EntitySeg Dataset Benchmarks

  • EntitySeg benchmarks semantic, instance, panoptic, and entity segmentation; COCO pretraining and stronger backbones improve performance, while EntitySeg is more challenging than COCO and ADE20K.For example, Swin-L Mask2Former has PQ 57.8% on COCO-PAN.Source: §5.1 · p. 9, Table 4 · p. 9
  • EntitySeg pretraining provides greater gains than COCO pretraining when transferred to image- and video-level segmentation tasks.The comparison reports relative performance gains after fine-tuning on other segmentation datasets.Source: §5.1 · p. 9, Table 5 · p. 9
  • On DIS5K, EntitySeg-pretrained Mask2Former outperforms task-specific ISNet despite having no task-specific design.The comparison follows fine-tuning on Dichotomous Image Segmentation.Source: §5.1 · p. 9
  • Without fine-tuning on evaluation datasets, EntitySeg-trained Mask2Former outperforms COCO-Panoptic and COCO-Entity on in-the-wild segmentation.The evaluation includes large-vocabulary, cluttered-indoor, camouflaged-instance, and few-shot segmentation.Source: §5.1 · p. 9

5.2 Comprehensive Analysis of CropFormer

CropFormer is evaluated through entity-segmentation ablations and transfer evaluations on other datasets.

5.2 Comprehensive Analysis of CropFormer

  • Combining CropFormer’s full-image and four-crop outputs yields a 1.5 APe gain over the Mask2Former baseline, with greater efficiency than simply increasing image resolution.Using CropFormer crop outputs alone also achieves a significant APe gain.Source: Table 6 · p. 10, §5.2 · p. 10
  • The transfer evaluation tests EntitySeg-pretrained Mask2Former after fine-tuning on the DIS5K dichotomous image segmentation task.The table caption identifies the compared model as pretrained on EntitySeg and then fine-tuned on DIS5K.Source: Table 7 · p. 10
  • The in-the-wild evaluation directly tests models trained on EntitySeg or COCO across LVIS, cluttered indoor, camouflaged-instance, and few-shot segmentation.The reported metric is AR@100 at IoU threshold 0.5.Source: Table 8 · p. 10

5.3 GELLA on Entity Grounding

GELLA is evaluated on referring expression segmentation, panoptic narrative grounding, and panoptic segmentation, with further studies of reasoning, zero-shot segmentation, and design choices.

5.3.1 Referring Expression Segmentation

  • GELLA outperforms GLAMM-13B and PixelLM on RefCOCO+ and RefCOCOg but falls short on RefCOCO; scaling from 7B to 13B does not improve referring accuracy.The authors suggest dataset annotation differences may explain the benchmark discrepancy.Source: Table 9 · p. 11, §5.3.1 · p. 11

5.3.2 Panoptic Narrative Grounding

  • With a 7B language branch, GELLA raises Average Recall (AR) by 3.7 points over PPO-TD on panoptic narrative grounding; a 13B branch improves performance further.The pipeline also outperforms GSVA and OMG-LLAVA on this task.Source: Table 10 · p. 11, §5.3.2 · p. 11

5.3.3 Panoptic Segmentation

  • GELLA improves COCO panoptic segmentation without a conventional Swin-Tiny image encoder, though its Swin-Large-derived colormaps make the comparison not entirely equitable.Increasing language-branch size does not improve segmentation performance.Source: Table 13 · p. 12, §5.3.3 · p. 12

5.3.4 Reasoning Segmentation

  • With a 13B language model, GELLA fits reasoning segmentation without ADE20K; the authors say its association module reduces dependence on large-scale semantic-aware data.Source: Table 14 · p. 12, §5.3.4 · p. 12

5.3.5 Zero-Shot Segmentation

  • On the SGinW zero-shot benchmark, GELLA outperforms Grounded-SAM and Grounded-HQ-SAM with large-scale training data, but exact training-set alignment is difficult.The benchmark covers 25 in-the-wild zero-shot datasets.Source: Table 14 · p. 12, §5.3.5 · p. 12

5.3.6 Ablation Studies

  • A 336-pixel CLIP vision input helps recognize small objects; centroid and random colormaps perform similarly, while the full ResoBlend configuration gives optimal performance.Removing any ResoBlend module degrades mask prediction quality.Source: §5.3.6 · pp. 12–13, Table 17 · p. 13
  • Colormap design outperforms region-feature and mask-weighted-attention alternatives; binary-mask encoder/decoder variants also underperform, with pooling and repeated passes among the cited drawbacks.The proposed single colormap lets the backbone capture entities jointly before attention.Source: §5.3.6 · pp. 13–14, Table 19 · p. 13
  • EntitySeg segmentation inputs outperform the other three for panoptic narrative grounding; a 0.2/0.2/0.6 sampling ratio yields the best entity-recognition and panoptic-narrative-grounding performance, while extra association-module FC layers add no improvement.The ratio is for image description, semantic noun extraction, and entity grounding, respectively.Source: §5.3.6 · pp. 14–15

5.4 Visualization

GELLA’s interface demonstrates image captioning, entity recognition, and panoptic narrative grounding by associating extracted nouns with entity masks.

5.4 Visualization

  • GELLA’s interface showcases image captioning, entity recognition, and panoptic narrative grounding.The model first describes the image in a long caption, then extracts semantic nouns and associates them with provided entity masks.Source: §5.4 · p. 15
  • GELLA first generates a long image caption, then extracts semantic nouns and associates them with the provided entity masks.Source: §5.4 · p. 15
  • The interface can also incorporate existing segmentation results as input.Source: §5.4 · p. 15

5.5 Failure Case

Pipeline performance depends on intermediate stages, and GELLA has failure cases in noun extraction, caption coverage, and part segmentation.

5.5 Failure Case

  • Final pipeline performance is influenced by entity segmentation, long-caption generation, entity recognition, and association.Source: §5.5 · p. 15
  • Semantic-noun extraction may miss ground-truth nouns, while generated captions may not account for every entity segmentation result.The paper gives omitted relational phrases such as “in front of the sofa” and “on the windowsill” as an example.Source: §5.5 · p. 15
  • GELLA currently cannot handle part segmentation or the subsequent association.The authors plan to address this in future work.Source: §5.5 · p. 15

6 CONCLUSION

ESG combines CropFormer and GELLA with EntitySeg to support high-quality entity segmentation and grounding. CropFormer fuses full-image and crop predictions, while GELLA uses language and visual components to associate masks with entities.

6 CONCLUSION

  • EntitySeg contains about 33K images from varied domains and resolutions, with high-quality mask annotations for training and testing.Source: §6 · p. 15
  • CropFormer fuses low-resolution full-image predictions with high-resolution crop predictions using batch-level queries.It is designed for the computational challenges of high-quality, high-resolution images.Source: §6 · p. 15
  • GELLA uses colormap and image encoders plus mask and language decoders for entity association.The colormap encoder prioritizes masks, allowing a CLIP vision encoder to handle low-resolution images in pixel-level prediction and semantic-noun extraction.Source: §6 · p. 15
  • ESG is intended as a flexible pipeline that can ground entities using offline masks from state-of-the-art segmentation methods.Source: §6 · p. 15
Loading 2402.02555v2…