High-Quality Entity Segmentation

1. Introduction

• EntitySeg addresses the lack of a suitable entity segmentation dataset with diverse images and high-quality masks for open-world study.The dataset prioritizes high-resolution imagery and high-quality annotations for pixel-precise editing, unlike existing panoptic datasets described as low-resolution and coarsely annotated.Source: §1 · p. 2
• High-resolution images and precise masks strain existing methods: resizing loses fine detail, direct processing is computationally intractable, and separate crops lose global context.Combining predictions across views is also unclear for Transformer methods based on set prediction.Source: §1 · p. 2
• CropFormer associates entities across the full image and high-resolution crops, then fuses their predictions to retain global context and fine-grained detail.Training fuses the full image with a randomly selected predefined corner crop; inference uses all four corner crops with the full image.Source: §1 · p. 2
• EntitySeg includes images from various domains and resolutions, while CropFormer and the dataset are evaluated across multiple segmentation tasks and settings.The authors report that over 70% of EntitySeg images are very high resolution, beyond what traditional training settings can accommodate.Source: §1 · p. 2

Image Segmentation Dataset

• Existing datasets cover semantic, instance, panoptic, and specialized tasks, but the paper identifies a need for diverse, high-quality images suited to real-world applications.EntitySeg spans indoor, outdoor, street, cartoon, and remote imagery; over 80% of its images fall in the 2000px–8000px range.Source: §2 · p. 3
• Unlike datasets with predefined category lists, EntitySeg annotates each entity class-agnostically before assigning category information in an open-vocabulary manner.The passage contrasts this approach with ADE20K and LVIS.Source: §2 · p. 3

Scene Parsing Methods

• Query-based methods such as Max-Deeplab and Mask2Former predict masks without boxes, but their object assignments can change across views.The passage attributes this limitation to set-prediction losses that do not enforce assignment constraints across views.Source: §2 · p. 3

3. EntitySeg Dataset

• EntitySeg contains 33,227 images, with 71.25% at least 2000px×2000px and 86.23% at least 1000px×1000px, open-world entities, and more precise boundary masks.Semantically coherent regions count as entities even when blurred or hard to recognize semantically.Source: §3 · p. 3

3.1. Image Collection

• Images come from public datasets and the Internet, with most sourced from high-resolution collections; EntitySeg includes resolutions up to 15,000px, unlike COCO and ADE20K images under 1000px.Sources include Pixabay, Unsplash, and LAION-400M, alongside datasets such as COCO and ADE20K.Source: §3.1 · p. 3

3.2. Image Annotation

• Annotation first creates non-overlapping class-agnostic pixel masks, then assigns open-vocabulary labels, including “unknown” or “blurred” when entities are hard to name or severely blurred.A “supercategory other” label is used when the supercategory is known but the fine-grained class is not.Source: §3.2 · p. 3
• EntitySeg has a greater proportion of covered image area and more masks on average than COCO-Panoptic and ADE20K-Panoptic, attributed to including all semantically coherent regions.The authors describe this annotation procedure as more similar to the class-agnostic human visual system.Source: §3.2 · p. 4, Table 2 · p. 5
• Reannotation of 500 images after four months showed high consistency in both class-agnostic mask localization and category labeling.The sample represents 1.5% of EntitySeg; the study used two additional annotators.Source: §3.2 · pp. 4–5

3.3. Dataset Statistics

• EntitySeg averages 18.1 entities per image, versus 11.2 in COCO and 13.6 in ADE20K, and its entities have more complex shapes than those datasets.The comparison covers average image resolution, entity counts, valid area, entity complexity and simplicity, and maximum per-image entity count.Source: §3.3 · p. 5, Table 2 · p. 5

CropFormer

• CropFormer fuses mask predictions from multiple views, using an association module to generate shared queries for the same entities across the full image and its crop.The full image supplies global context while crops provide fine-grained local details.Source: §4 · pp. 5–6
• The crop dataloader pairs each full image with a fixed-ratio crop from one of four corners and resizes both views to the same dimensions.Fixed corner cropping keeps training and inference consistent, with a fixed number of crops sufficient at inference.Source: §4 · p. 5
• The image-level decoder produces embeddings for the full image and crop, which support image-level entityness and pixel-level mask predictions.Image-level queries are separate for each view.Source: §4 · p. 6
• The batch-level decoder derives embeddings that are shared between the full image and crop, unlike the separate queries used by the image-level decoder.The shared embeddings and low-level batch features are used for batch-level mask classification and pixel-wise prediction.Source: §4 · p. 6
• Training uses separate image- and batch-level losses, while inference averages mask predictions across the full image and all four corner crops.The final entity confidence comes from the batch-level entityness prediction.Source: §4 · pp. 6–7

5.1. Dataset Benchmarks

• EntitySeg benchmarks cover semantic, instance, panoptic, and entity segmentation; results are more challenging than COCO/ADE20K, while COCO pretraining and stronger backbones help.Source: §5.1 · p. 7, Table 3 · p. 7
• EntitySeg pretraining yields greater improvements than COCO pretraining when transferred to other image- and video-level segmentation datasets.Source: §5.1 · p. 7, Table 4 · p. 7
• On DIS5K, EntitySeg-pretrained then finetuned Mask2Former outperforms task-designed ISNet by a large margin, attributed to EntitySeg’s high-quality masks.Source: §5.1 · p. 7, Table 5 · p. 7
• Without finetuning, EntitySeg-trained Mask2Former outperforms COCO-Panoptic and COCO-Entity by large margins on in-the-wild evaluations.Source: §5.1 · p. 7, Table 7 · p. 8

5.2. A Comprehensive Analysis of CropFormer

• Combining CropFormer’s full-image and four-crop outputs yields a 1.5 APe gain over the baseline, while remaining more efficient than naive resolution increases.Table 6 compares CropFormer with Mask2Former and related multi-view approaches.Source: Table 6 · p. 8, §5.2 · p. 8
• CropFormer improves over Mask2Former with 2×/3× schedules; crop-only decoding beats full-image-only decoding, and combining both views works best.Source: §5.2 · p. 8, Table 8 · p. 8
• A crop ratio of 0.7 and fixed 4/8 crops perform best; self-attention versus feedforward after cross-attention yields only slight performance differences.Source: §5.2 · pp. 8–9, Table 9 · p. 9
• CropFormer significantly exceeds Mask Transfiner and PatchDCT on traditional and boundary AP, and achieves competitive instance and panoptic gains on COCO and ADE20K.Source: §5.2 · p. 9, Table 10 · p. 9

• EntitySeg contains about 33K images from varied domains and resolutions, with high-quality mask annotations for training and testing.Source: §6 · p. 9
• CropFormer uses batch-level queries to fuse predictions from a low-resolution full image and its high-resolution crops, addressing computation challenges posed by EntitySeg.Source: §6 · p. 9
• The authors hope EntitySeg and CropFormer will catalyze future work on open-world, high-quality image segmentation.Source: §6 · p. 9