Source-linked AI summary
FocalClick: Towards Practical Interactive Image Segmentation
Xi Chen, Zhiyan Zhao, Yilei Zhang, Manni Duan, Donglian Qi, Hengshuang Zhao
TL;DR
Interactive segmentation still faces efficiency limits on low-power devices and difficulty correcting preexisting masks without destroying correct regions. FocalClick uses localized coarse and refinement predictions plus Progressive Merge, and reports improved efficiency, competitive SOTA performance, and superiority on mask correction.
Problem
Existing interactive segmentation methods are insufficiently efficient for low-power deployment and perform poorly when refining preexisting masks.
Method
FocalClick predicts on localized Target and Focus Crops and uses Progressive Merge to update intended regions while preserving other mask details.
Results
FocalClick reaches SOTA with significantly smaller FLOPs and shows significant superiority on interactive mask correction.
Takeaways & Limitations
FocalClick improves efficiency and compatibility with existing annotation pipelines, including workflows that begin with preexisting masks.
Takeaways & Limitations
Performance on tiny structures remains unsatisfactory, and 4K images introduce loading, movement, zoom, and visualization bottlenecks requiring engineering effort.
Abstract
from arXiv · showhide
Interactive segmentation allows users to extract target masks by making positive/negative clicks. Although explored by many previous works, there is still a gap between academic approaches and industrial needs: first, existing models are not efficient enough to work on low power devices; second, they perform poorly when used to refine preexisting masks as they could not avoid destroying the correct part. FocalClick solves both issues at once by predicting and updating the mask in localized areas. For higher efficiency, we decompose the slow prediction on the entire image into two fast inferences on small crops: a coarse segmentation on the Target Crop, and a local refinement on the Focus Crop. To make the model work with preexisting masks, we formulate a sub-task termed Interactive Mask Correction, and propose Progressive Merge as the solution. Progressive Merge exploits morphological information to decide where to preserve and where to update, enabling users to refine any preexisting mask effectively. FocalClick achieves competitive results against SOTA methods with significantly smaller FLOPs. It also shows significant superiority when making corrections on preexisting masks. Code and data will be released at github.com/XavierCHEN34/ClickSEG
1. Introduction
FocalClick addresses efficiency and preexisting-mask correction by focusing computation and updates on localized regions rather than recomputing or overwriting the whole image.
- Click-based interactive segmentation lets users define foreground and background through successive positive and negative clicks.
- Existing methods struggle on low-power devices because reducing input resolution can significantly reduce accuracy.They often prioritize accuracy with large models and high-resolution inputs.
- Existing methods also handle preexisting masks poorly, sometimes failing to remove errors and changing regions far from user clicks.
- Previous pipelines recompute all pixels after each click, causing redundant computation and risking destruction of correct mask details.
- FocalClick predicts on noteworthy patches and updates only the region the user intends to modify, retaining other mask regions.
- Its efficient pipeline uses a low-resolution Target Crop for coarse segmentation and a local Focus Crop for refinement, replacing full-image inference with two local predictions.
- Progressive Merge uses morphological analysis to decide where to update or preserve, supporting correction of preexisting masks without destroying correct parts.
2. Related Work
FocalClick builds on local and coarse-to-fine interactive segmentation while selecting refinement regions more selectively to reduce computation and support localized correction.
- Earlier click-based methods established image-plus-click-map prediction, while later work addressed ambiguity through multiple candidate masks or selection mechanisms.
- Some methods crop around the last predicted target, but they retain crop resolution because they predict final masks directly on that crop.
- FocalClick uses a Target Crop to locate a Focus Crop and relies on local refinement rather than requiring the Segmentor to produce fine details.
- Other coarse-to-fine methods refine click-based regions or primitive boundaries, generally obtaining finer results at the cost of larger computation.
3. Method
FocalClick decomposes segmentation into target localization, coarse prediction, focused refinement, and selective merging, while introducing a benchmark for correcting preexisting masks.
- FocalClick decomposes full-image inference into two light predictions on small patches to bridge practical deployment needs.
- Efficient pipeline: Target Crop expands around the previous mask and new click, then crops and resizes the image, mask, and click maps for coarse segmentation.
- Efficient pipeline: The Segmentor predicts a rough target mask that helps locate the region requiring detailed refinement.
- Efficient pipeline: Focus Crop selects the maximum connected difference region containing the new click and expands it for local refinement.
- Efficient pipeline: The Refiner combines detail and boundary predictions with coarse logits to produce a refined local prediction.
- Progressive Merge: Progressive Merge updates only the maximum connected difference region containing the click and leaves the previous mask unchanged elsewhere.
- Interactive Mask Correction: Interactive Mask Correction addresses preexisting masks, and DAVIS-585 was constructed to evaluate this ability.
- Interactive Mask Correction: DAVIS-585 samples separate object annotations from DAVIS2017 videos, filters masks under 300 pixels, and contains 585 test samples.
4. Experiment
Experiments evaluate FocalClick’s configurations, efficiency, benchmark performance, mask correction, ablations, and qualitative behavior. The results show competitive segmentation with substantially lower computation, improved correction from preexisting masks, and benefits from its localized components.
- Experimental Configuration: FocalClick supports S1 models for edge devices and web browsers and S2 models for CPU laptops.
- Computation Analysis: 15 times smaller FLOPs than the lightest RITM and 360 times smaller than FCANet are achieved by FocalClick B0-S1.Using the same Segmentor, hrnet-18s reduces FLOPs by 2 to 8 times compared with original RITM.
- Comparison with State-of-the-Art: Competitive or superior performance against previous SOTA methods is achieved with significantly smaller FLOPs across FocalClick versions.With a large combined training dataset, FocalClick outperforms other methods by a large margin.
- Performance for Mask Correction: FocalClick requires significantly fewer clicks than starting from scratch on preexisting masks, while its S1 version outperforms big RITM with 1/67 FLOPs.The DAVIS-585 initial masks have IoUs between 0.75 and 0.85, with some challenging details already annotated.
- Ablation Study: Adding Target Crop, Focus Crop, and Progressive Merge steadily improves annotation from both initial masks and scratch.The naive full-image pipeline performs especially poorly at the small S1 resolution, while the modules compensate for the disadvantage of small inputs.
- Qualitative Result: The Refiner recovers fine details, while qualitative results show high-quality predictions within several clicks and preservation of well-segmented details during correction.A failure case involves tiny parachute-rope structures, for which users can zoom in and switch to manual brush editing.
5. Limitation
FocalClick improves efficiency and compatibility with existing annotation pipelines, but its performance on tiny structures remains unsatisfactory and high-resolution images introduce new speed bottlenecks.
- Tiny-structure performance remains unsatisfactory and could be improved with more finely annotated data or matting datasets.
- For 4K images, image loading, data movement, zooming, and visualization become new speed bottlenecks.These costs can be ignored for images under 1080P.
- Building a practical annotation system still requires extensive engineering efforts.
6. Conclusion
FocalClick addresses practical problems in interactive segmentation by improving efficiency and introducing interactive mask correction for real-world requirements.
- FocalClick targets practical problems in interactive segmentation.
- The method significantly improves pipeline efficiency for deployment on low-power devices.
- It formulates interactive mask correction as a new task to meet real-world requirements.