Source-linked AI summary
Bilateral Reference for High-Resolution Dichotomous Image Segmentation
Peng Zheng, Dehong Gao, Deng-Ping Fan, Li Liu, Jorma Laaksonen, Wanli Ouyang, Nicu Sebe
TL;DR
High-resolution dichotomous image segmentation still struggles with fine details. BiRefNet combines localization and reconstruction with bilateral image and gradient references, and it reports state-of-the-art performance across DIS5K, HRSOD, and COD while adding practical training strategies.
Problem
Existing high-resolution segmentation strategies remain insufficient for capturing very fine features, especially where object regions resemble the background in color and texture.
Method
BiRefNet uses separate localization and reconstruction modules, bilateral source-image and gradient references, auxiliary gradient supervision, and DIS-specific training strategies.
Results
BiRefNet achieves state-of-the-art performance across DIS5K, HRSOD, and COD, with average Sm improvements of 6.8%, 2.0%, and 5.6%, respectively.
Takeaways & Limitations
The framework transfers across dichotomous, high-resolution salient, and concealed object detection tasks and supports unified high-resolution segmentation.
Abstract
from arXiv · showhide
We introduce a novel bilateral reference framework (BiRefNet) for high-resolution dichotomous image segmentation (DIS). It comprises two essential components: the localization module (LM) and the reconstruction module (RM) with our proposed bilateral reference (BiRef). The LM aids in object localization using global semantic information. Within the RM, we utilize BiRef for the reconstruction process, where hierarchical patches of images provide the source reference and gradient maps serve as the target reference. These components collaborate to generate the final predicted maps. We also introduce auxiliary gradient supervision to enhance focus on regions with finer details. Furthermore, we outline practical training strategies tailored for DIS to improve map quality and training process. To validate the general applicability of our approach, we conduct extensive experiments on four tasks to evince that BiRefNet exhibits remarkable performance, outperforming task-specific cutting-edge methods across all benchmarks. Our codes are available at https://github.com/ZhengPeng7/BiRefNet.
1. Introduction
The paper targets high-resolution dichotomous image segmentation, where existing strategies remain insufficient for very fine features. BiRefNet addresses this with separate localization and reconstruction modules, bilateral references, auxiliary supervision, and practical training strategies.
- High-resolution DIS requires precise segmentation of salient or concealed objects, including fine structures in high-resolution images.
- Existing intermediate-supervision, frequency-prior, and unite-divide-unite strategies remain insufficient to capture very fine features.
- BiRefNet separates object localization from reconstruction, using hierarchical features for coarse low-resolution predictions and reference-guided decoding for refinement.
- Its bilateral reference module combines source-image guidance as an inward reference with gradient supervision as an outward reference for reconstructing high-resolution results.
- The paper also summarizes DIS-specific strategies intended to improve performance, prediction quality, and convergence acceleration.
- BiRefNet reports state-of-the-art performance across DIS5K, HRSOD, and COD, with average Sm improvements of 6.8%, 2.0%, and 5.6%, respectively.
2. Related Works
Related work addresses high-resolution class-agnostic segmentation through task-specific datasets, progressive refinement, auxiliary priors, and memory-efficient processing. BiRefNet instead uses intact high-resolution images with bilateral references within a localization–reconstruction framework.
- High-resolution class-agnostic segmentation includes DIS, high-resolution salient object detection, and concealed object detection.
- High-resolution DIS emphasizes complex slender target structures, making it more challenging than ordinary segmentation settings.
- Prior methods use intermediate supervision, frequency priors, label decoupling, or image patches to improve detail capture or training efficiency.
- BiRefNet differs by using intact source images at original scales and gradient priors as bilateral references within separate localization and reconstruction modules.
- Progressive-refinement approaches guide higher-resolution predictions using scaled images, predicted maps, refiners, aligned features, or auxiliary information.
3. Methodology
BiRefNet separates high-resolution DIS into localization and reconstruction, using bilateral references to preserve image detail and emphasize fine structures. Its training combines multi-level losses and multi-stage supervision to improve detail quality and convergence efficiency.
- Localization and reconstruction: BiRefNet decomposes high-resolution DIS into a localization module for semantic target localization and a reconstruction module for refining predictions.The localization pathway uses hierarchical encoder features, classification-based semantic representation, bottleneck squeezing, and ASPP-based multi-context fusion.
- Reconstruction module: Reconstruction blocks use deformable convolutions with hierarchical receptive fields to balance broad context for localization against detailed feature extraction.Features from different receptive fields are concatenated and transformed before producing reconstruction-module outputs.
- Bilateral reference: The inward reference adaptively crops original-resolution image patches and supplies them to decoder stages alongside features, preserving high-resolution information throughout reconstruction.Unlike approaches that add the source image only at the final stage or resize it, adaptive cropping matches patches to decoder-stage resolutions.
- Bilateral reference: The outward reference uses gradient labels and auxiliary gradient prediction to emphasize regions with dense gradient information that are important for fine-structure segmentation.Gradient-aware features are converted into attention that acts on original features; masking removes gradients outside dilated intermediate-prediction regions.
- Training strategies: The objective combines BCE, IoU, SSIM, and CE losses to supervise pixel, region, boundary, and semantic properties, while multi-stage supervision accelerates training.With MSS, 200 epochs can achieve similar performance to 400 epochs, cutting training time in half; longer training mainly improves fine-detail metrics at high computational cost.
4. Experiments
BiRefNet is evaluated across DIS, HRSOD, COD, and SOD using established datasets, metrics, ablations, and comparisons with task-specific models. The experiments show gains from its reconstruction and bilateral-reference components, practical training strategies, and cross-task generalization.
- Evaluation protocol: BiRefNet is evaluated on DIS5K, HRSOD, COD, and supplementary low-resolution SOD test sets using S-measure, F-measures, E-measure, MAE, and HCE.The evaluation includes all DIS5K test sets, three HRSOD sets, three COD sets, and two low-resolution SOD sets.
- Ablation study: The reconstruction module provides multi-scale receptive fields and yields approximately 2.2% relative improvement in Fβ with little extra computational cost.The module targets local details while retaining overall semantics in high-resolution features.
- Ablation study: The inward and outward references jointly provide a 2.9% relative improvement in Fβ, while combining RM and BiRef reaches 6.2% relative improvement.InRef supplies lossless high-resolution information, whereas OutRef emphasizes fine-detail regions through gradient labels.
- Training strategies: The proposed training strategies improve different aspects of performance: CFF and IPT improve overall results, while RLFT targets edge-detail precision.These strategies include context feature fusion, image-pyramid input, and regional-loss fine-tuning.
- State-of-the-art comparison: BiRefNet achieves leading results across DIS, HRSOD, COD, and SOD and improves average S-measure by 2.0% on HRSOD and 5.6% across three COD benchmarks.The cross-task comparison uses the best task-specific models for each benchmark family.
- State-of-the-art comparison: BiRefNet outperforms previous methods in global shape and pixel details, including distractor rejection, slim shapes, curved edges, and fragmented occluded objects.Qualitative comparisons report more accurate segmentation on DIS and COD samples with neighboring distractors, fine structures, and occlusions.
5. Potential Applications
BiRefNet is presented as useful for fine-grained segmentation applications, including crack detection and high-accuracy object extraction. The examples emphasize complex shapes, thin structures, and detailed boundaries.
- Crack Detection: BiRefNet can detect wall cracks and support maintenance by identifying fine, complex structures.The paper contrasts this with models trained on common datasets that mainly segment regular foreground objects.
- Highly Accurate Object Extraction: BiRefNet can extract complex foreground objects and remove backgrounds without manual masks.The paper reports higher-resolution results that can segment hair-level thin threads.
6. Third-Party Creations
BiRefNet has been incorporated into community-built applications, including ComfyUI integrations and an online Fal.AI demo. Community use also includes social-media comparisons with other segmentation systems.
- Practical Applications: Developers integrated BiRefNet into ComfyUI to support foreground matting and subsequent Stable Diffusion processing.The integration is described as a node for improved downstream processing.
- Practical Applications: Fal.AI provides an online BiRefNet demo on an A6000 GPU with prediction and HTTP API access.The service is presented as a way to access the model online.
- Social Media: Community tests reported competitive BiRefNet results against BRIA RMGB v1.4.The comparison is described as appearing in social-media tests.
7. Conclusions
The paper concludes that bilateral references improve fine-detail capture across dichotomous, salient-object, and concealed-object segmentation. It also presents practical training strategies and reports strong generalization across 13 benchmarks, alongside continued transfer into applications.
- Conclusions: BiRefNet uses inward and outward references to restore missing fine information and emphasize detail-rich regions.The paper links these references to improved capture of tiny-pixel features.
- Conclusions: The framework supports dichotomous, high-resolution salient-object, and concealed-object segmentation within one model.The conclusion describes this as a unified framework across these tasks.
- Conclusions: The paper reports competitive results and strong generalization on 13 benchmarks.It also describes practical tricks intended to improve prediction quality and convergence speed for high-resolution training.
- Conclusions: The techniques can transfer to practical applications and may encourage unified models and developer-created tools.The paper explicitly connects the framework to broader academic and developer-community use.