Source-linked AI summary
Highly Accurate Dichotomous Image Segmentation
Xuebin Qin, Hang Dai, Xiaobin Hu, Deng-Ping Fan, Ling Shao, Luc Van Gool
TL;DR
Existing segmentation tasks and datasets provide limited support for highly accurate, category-agnostic segmentation of objects with diverse structures and high-resolution details. The paper introduces DIS5K, IS-Net, and HCE, then benchmarks 16 models and reports that DIS is solvable with applications including background removal and 3D reconstruction. The authors also identify remaining dataset-diversity and HCE-related limitations.
Problem
Existing segmentation datasets and tasks often have limited resolution or geometric accuracy and focus on specific characteristics or categories rather than highly accurate, category-agnostic objects.
Method
The paper constructs the DIS5K dataset, introduces IS-Net with intermediate supervision, and designs HCE to measure human correction efforts for faulty predictions.
Results
The authors conclude that DIS is solvable, with IS-Net achieving high-quality segmentation and the benchmark showing promising results for background removal and 3D reconstruction.
Takeaways & Limitations
DIS5K, IS-Net, and HCE provide a foundation for studying highly accurate segmentation and its practical relationship to human correction effort.
Takeaways & Limitations
The dataset needs more categories, samples, and diversified image qualities, while HCE depends on erosion and dilation to remove small faulty regions under chosen accuracy requirements.
Abstract
from arXiv · showhide
We present a systematic study on a new task called dichotomous image segmentation (DIS) , which aims to segment highly accurate objects from natural images. To this end, we collected the first large-scale DIS dataset, called DIS5K, which contains 5,470 high-resolution (e.g., 2K, 4K or larger) images covering camouflaged, salient, or meticulous objects in various backgrounds. DIS is annotated with extremely fine-grained labels. Besides, we introduce a simple intermediate supervision baseline (IS-Net) using both feature-level and mask-level guidance for DIS model training. IS-Net outperforms various cutting-edge baselines on the proposed DIS5K, making it a general self-learned supervision network that can facilitate future research in DIS. Further, we design a new metric called human correction efforts (HCE) which approximates the number of mouse clicking operations required to correct the false positives and false negatives. HCE is utilized to measure the gap between models and real-world applications and thus can complement existing metrics. Finally, we conduct the largest-scale benchmark, evaluating 16 representative segmentation models, providing a more insightful discussion regarding object complexities, and showing several potential applications (e.g., background removal, art design, 3D reconstruction). Hoping these efforts can open up promising directions for both academic and industries. Project page: https://xuebinqin.github.io/dis/index.html.
MBZUAI Abu Dhabi, UAE
The paper formulates category-agnostic dichotomous image segmentation for highly accurate objects with diverse structures, addressing limitations in existing task-specific datasets and metrics. It introduces DIS5K, IS-Net, HCE, and a broad benchmark to connect segmentation quality with practical applications.
- Existing segmentation tasks usually target specific characteristics or categories and are often dataset-dependent.
- DIS5K provides 5,470 high-resolution images with highly accurate binary segmentation masks.
- IS-Net uses intermediate supervision to reduce over-fitting through direct feature synchronization in high-dimensional feature spaces.
- HCE measures the human interventions needed to correct faulty predictions, complementing conventional mathematical or cognitive metrics.
- The benchmark compares IS-Net with 16 cutting-edge segmentation models and reports applications in background removal and 3D reconstruction.
3. Proposed DIS5K Dataset
DIS5K is a large, manually annotated high-resolution dataset designed to cover diverse object categories and structural complexities for category-agnostic segmentation. Its analysis shows substantially greater complexity and diversity than existing datasets, with evaluation splits organized by shape complexity.
- Data Collection and Annotation: DIS5K contains 5,470 images across 225 categories and 22 groups, selected to cover complicated structures and reduce labeling confusion.Most selected images contain single objects, while categories include common but difficult targets such as fences, stairs, cables, bonsai, and trees.
- Data Collection and Annotation: Each DIS5K image has a pixel-wise binary mask manually annotated in GIMP, averaging about 30 minutes per image and reaching up to 10 hours for difficult cases.The masks target foreground objects regardless of whether they are salient, camouflaged, meticulous, or otherwise characterized.
- Data Collection and Annotation: DIS5K is organized around category-based retrieval but supports category-agnostic segmentation through diversified textures, structures, shapes, contrasts, and complexities.The paper treats categories as a way to guarantee dataset diversity and facilitate tracing or extending collected samples, not as the segmentation task definition.
- Data Analysis: DIS5K compares 5,470 high-resolution images with 19 related datasets using image number, image dimensions, and object-complexity measures including IPQ, Cnum, and Pnum.The comparison includes salient, high-resolution salient, camouflaged, semantic, and thin-object segmentation datasets.
- Data Analysis: DIS5K includes greater intra-categorical complexity diversity than comparison datasets and avoids synthetic-data biases by using real images with varied structures.The dataset spans examples from relatively simple regional components to thin, intricate structures with high IPQ and Pnum.
- Data Analysis: DIS5K is around 20 times, and up to 50 times, more complex than SOD datasets by average IPQ, while its average Cnum exceeds 100 and its average Pnum exceeds 1,400.These measures reflect overall structural complexity, contour-level detail, and dominant boundary points, respectively.
- Data Splits: The 5,470 images are split into DIS-TR (3,000), DIS-VD (470), and DIS-TE (2,000), with the test set further divided into four subsets of ascending shape complexity.This design supports evaluation across progressively more difficult object shapes and structures.
4. Proposed IS-Net Baseline
IS-Net combines a ground-truth encoder, an image segmentation component, and intermediate supervision to synchronize high-dimensional features and improve segmentation of fine structures at large resolutions.
- Architecture: IS-Net contains a 27.7 MB ground-truth encoder, a 176.6 MB image segmentation component, and an intermediate supervision strategy.The ground-truth encoder encodes masks into high-dimensional spaces, while the segmentation component handles fine structures and large inputs.
- Ground-truth encoder: The ground-truth encoder is trained from segmentation masks with binary cross-entropy supervision across D intermediate feature maps.Its weights are then frozen to generate high-dimensional ground-truth features for supervising the segmentation model.
- Intermediate supervision: Intermediate supervision synchronizes corresponding high-dimensional features from the image segmentation model and the ground-truth encoder using a feature-consistency loss.Each segmentation feature map has the same dimensions as its corresponding ground-truth feature map.
- Intermediate supervision: The segmentation model also uses binary cross-entropy losses on side outputs, with weights assigned to the individual side-output losses.This combines feature-level supervision with mask-level side-output supervision.
- Feature representations: Ground-truth encoder channels represent complementary object structures, including fine, middle-sized, delicate, and large-scale regions.The reported channels encode bicycle structures such as frames, seats, wheels, brake cables, spokes, and overall shape.
5. Proposed HCE Metric
HCE evaluates segmentation predictions by approximating the human effort needed to correct false-positive and false-negative regions, including under a configurable tolerance for minor errors.
- Motivation: HCE complements conventional mathematical or cognitive consistency metrics by measuring the practical correction cost of prediction errors.Existing metrics include IoU, boundary IoU, F-measure, boundary F-measure, and MAE.
- Metric definition: Human Correction Efforts (HCE) approximates the number of mouse clicks required to correct faulty predictions for real-world accuracy requirements.It models boundary-point selection and region selection as the two frequently used correction operations.
- Metric definition: HCE separately considers false-positive and false-negative regions and categorizes them by their adjacent true or false regions.The categories include FNN, FNTP, FPP, and FPTN regions.
- Relaxed HCE: HCEγ introduces an error tolerance γ by excluding small false-positive and false-negative components through erosion and dilation.The example γ = 5 denotes the size of small faulty regions to ignore.
- Relaxed HCE: Skeleton information from the ground-truth mask is combined with the relaxed false-negative mask to retain thin structures that erosion could remove.This preserves critical components such as thin cables and nets regardless of their sizes.
6. DIS5K Benchmark
The DIS5K benchmark evaluates segmentation models on diverse, high-complexity objects using multiple metrics and fair dataset splits. IS-Net performs competitively across metrics, while ablations highlight the importance of spatial detail and intermediate supervision.
- DIS5K uses separate training, validation, and test splits, while cross-dataset evaluation is omitted because existing labels are not accuracy-consistent.
- The benchmark evaluates models with six metrics, including F-measure, mean absolute error, structural and enhanced alignment measures, and HCE.
- Quantitative Evaluation: IS-Net achieves the most competitive performance across all metrics against 16 state-of-the-art models.
- Quantitative Evaluation: DIS5K’s non-convex, fine structures require models to preserve spatial information, which many conventional backbones reduce through early downsampling.
- Qualitative Evaluation: Qualitative comparisons show promising IS-Net results for salient, camouflaged, thin, and meticulous objects across diverse scenes.
- Ablation Studies: Intermediate supervision improves performance over U2-Net and its adapted variant across random seeds, with Last-6 decoder-stage supervision selected as the default.
7. Conclusions
The paper formulates DIS, builds DIS5K, introduces IS-Net and HCE, and benchmarks segmentation models and applications. Its experiments support the conclusion that highly accurate DIS is solvable and relevant to academic and industrial applications.
- The study introduces DIS5K, IS-Net, and HCE as dataset, model, and evaluation contributions for highly accurate dichotomous segmentation.
- Comprehensive ablations and benchmarking support the conclusion that the newly formulated DIS task is solvable.
- The work reports potential relevance to segmentation applications in academia and industry.
-Supplementary Material
The supplementary material includes an artistic-figure demonstration generated from a DIS5K sample.
- An artistic figure is generated from a sample of the DIS5K dataset.
1. Related Work
The related work motivates DIS as a category-agnostic task for highly accurate segmentation of objects with diverse structures. Existing datasets and metrics often provide limited resolution, structural diversity, or practical relevance.
- Multi-class vs. Dichotomous Segmentation: Many practical applications require accurate segmentation of particular objects regardless of category, often in high-resolution images with few occlusions.
- Multi-class vs. Dichotomous Segmentation: Category-agnostic DIS targets highly accurate segmentation of objects with diversified shapes and structures.
- Datasets: Existing segmentation datasets commonly trade off geometrical labeling accuracy, image resolution, object complexity, or structural detail.
- Models: Deep segmentation models can fit training sets closely while showing different test performance because of distribution differences and architectural feature representations.
- Evaluation Metrics: Conventional metrics measure prediction consistency but do not reflect the practical effort required to correct faulty regions in applications.
2. More Details of DIS5K Dataset
DIS5K contains diverse, high-resolution objects spanning many categories and structural complexities. Its design supports analysis of both within-category and across-category structure similarities.
- 5,470 images span 225 categories organized into 22 groups, with average counts of about 24 images per category and 249 per group.
- DIS5K includes characteristics associated with salient, cluttered, camouflaged, thin, and meticulous object segmentation tasks.
- Object structure analysis: Objects within one category often share structures, but variations arise from missing components, design changes, viewpoints, and multiple targets.
- Object structure analysis: Different categories can also have similar structures, such as cracks, lightning, cables, ropes, and pipes composed of thin elongated components.
- Testing subsets: The four testing subsets are arranged in ascending structural complexity while maintaining similar image dimensions.
3. More Details of Experiments
Experiments compare IS-Net and segmentation baselines across validation and complexity-ordered test sets. The results show strong overall performance, while HCE and intermediate supervision expose aspects that conventional metrics and training curves miss.
- Experimental setup: Models are trained on DIS-TR with 3,000 images and validated on DIS-VD with 470 images after offline resizing to 1024 × 1024.
- Performance comparisons: IS-Net achieves the most competitive performance against existing models on almost all evaluation metrics across datasets.
- Complexity-aware evaluation: Conventional metrics can misleadingly suggest that DIS-TE4 is less challenging than DIS-TE1, whereas HCEγ ↓ tracks their structural-complexity difference.
- Complexity-aware evaluation: HCEγ ↓ rises from 149 on DIS-TE1 to 2,888 on DIS-TE4, indicating roughly 20 times greater correction cost for the more complex subset.
- Intermediate supervision: Intermediate supervision accelerates convergence before about 10,000 iterations and significantly decreases validation mean absolute error M ↓.
4. Applications
DIS5K supports applications that require accurate object masks, including editing, background removal, downstream segmentation, shape analysis, and 3D modeling.
- Editing and design: DIS5K provides materials for art design, image and video editing, posters, slides, and device wallpapers.
- Downstream segmentation: DIS5K can provide pretrained segmentation models and subsets for downstream tasks and rapid prototyping.
- Related vision tasks: The dataset can support high-resolution classification, segmentation-guided super-resolution, image processing, composite-image synthesis, and contour detection.
- Background removal: Background-removed images generated from DIS5K are described as aesthetically valuable and usable for direct editing applications.
- 3D and structure applications: Its masks can support 3D modeling, shape and structure analysis, and topology research.
5. Limitations and Future Works
The model still struggles with cluttered backgrounds, extremely thin structures, and low-contrast targets. The dataset and HCE metric also have scope and efficiency limitations that motivate future work.
- Model failure cases: The model fails on ship masts and ropes against cluttered backgrounds, and on one-pixel-wide mesh structures in 1024 × 1024 inputs.
- Model failure cases: Low color contrast between a key chain and its background further increases segmentation difficulty.
- Dataset limitations: The DIS5K dataset needs more categories, category-specific samples, and diversified image qualities to improve its diversity.
- Metric limitations: HCE evaluation is time-consuming because skeleton extraction and weighted F-measure computation rely on costly processing for large masks.