Source-linked AI summary
f-BRS: Rethinking Backpropagating Refinement for Interactive Segmentation
Konstantin Sofiiuk, Ilia Petrov, Olga Barinova, Anton Konushin
TL;DR
Interactive segmentation must reduce costly annotation while handling cases where deep networks underuse clicks or fail to improve with additional input. f-BRS reformulates BRS optimization over auxiliary feature parameters and restricts backpropagation to a small network portion. Across four standard benchmarks, it reports state-of-the-art speed and accuracy, with faster refinement than full-network BRS.
Problem
Interactive segmentation remains limited by high annotation costs, semantic overreliance, and nonconvergent feed-forward results on some images despite many clicks.
Method
f-BRS reparameterizes interactive-segmentation optimization over auxiliary parameters for intermediate features, so forward and backward passes run through only a small network part.
Results
f-BRS achieves state-of-the-art speed and accuracy on GrabCut, Berkeley, DAVIS, and SBD, and reduces hard failure cases by several times with significant NoC100 improvement.
Takeaways & Limitations
Feature refinement lets BRS adapt network outputs to user input without explicitly passing clicks to the network while avoiding whole-network backward computation.
Takeaways & Limitations
Experiments use the standard DeepLabV3+ architecture rather than focusing on architecture improvements, and the formulation assumes λ is constant in later works.
Abstract
from arXiv · showhide
Deep neural networks have become a mainstream approach to interactive segmentation. As we show in our experiments, while for some images a trained network provides accurate segmentation result with just a few clicks, for some unknown objects it cannot achieve satisfactory result even with a large amount of user input. Recently proposed backpropagating refinement (BRS) scheme introduces an optimization problem for interactive segmentation that results in significantly better performance for the hard cases. At the same time, BRS requires running forward and backward pass through a deep network several times that leads to significantly increased computational budget per click compared to other methods. We propose f-BRS (feature backpropagating refinement scheme) that solves an optimization problem with respect to auxiliary variables instead of the network inputs, and requires running forward and backward pass just for a small part of a network. Experiments on GrabCut, Berkeley, DAVIS and SBD datasets set new state-of-the-art at an order of magnitude lower time per click compared to original BRS. The code and trained models are available at https://github.com/saic-vul/fbrs_interactive_segmentation .
1. Introduction
Interactive segmentation reduces annotation burden but remains vulnerable to semantic overreliance and expensive refinement. f-BRS reparameterizes BRS over auxiliary feature variables, retaining refinement while limiting backpropagation to a small network part.
- Motivation: Object segmentation supports fine-grained scene representation, but its practical use is limited by extremely high annotation costs.Large-scale annotation is needed because deployed systems continually generate new data requiring labeling.
- Problem: Deep learning-based interactive segmentation can overuse image semantics and make little use of user clicks.Training clicks often correspond perfectly with image semantics, allowing networks to downweight them.
- Problem: BRS improves hard-case refinement by enforcing mask consistency with clicks, but repeatedly backpropagates through the whole model.This substantially increases the computational budget per click compared with other methods.
- Contribution: f-BRS optimizes auxiliary parameters applied to intermediate features, requiring forward and backward passes only through the network’s last several layers.The auxiliary parameters are invariant to image position, avoiding the insufficient receptive field of directly optimized late activations.
- Results: Experiments on GrabCut, Berkeley, DAVIS, and SBD report state-of-the-art speed and accuracy.The paper evaluates the proposed refinement scheme across four standard interactive-segmentation benchmarks.
2. Related work
Prior interactive-segmentation methods balance predictable optimization with semantic deep learning, but each has shortcomings in user-input use or computational cost. BRS combines both paradigms, while f-BRS shifts optimization from network inputs to auxiliary parameters for faster refinement.
- Interactive segmentation: Interactive segmentation commonly uses positive and negative clicks, with classical methods framing mask generation as an optimization problem.Classical approaches rely on hand-designed heuristics and require substantial user input.
- Optimization-based methods: Optimization-based methods provide predictable behavior and detailed masks with enough input but make insufficient use of semantic priors.This can require additional effort even for objects that learning-based methods recognize well.
- Learning-based methods: Deep learning-based methods often overweight image semantics because training clicks align closely with image semantics.As a result, user-provided clicks may contribute little information during training and be downweighted.
- Backpropagating refinement: BRS finds minimal edits to click distance maps that produce masks consistent with user annotation.Its refinement relies on optimizing network behavior while keeping the network weights fixed.
- Feature backpropagating refinement: f-BRS optimizes auxiliary parameters instead of network inputs and is reported as an order of magnitude faster than BRS.The reparameterization avoids forward and backward passes through the whole network while preserving the refinement objective.
3. Proposed method
The method reformulates interactive-segmentation refinement to optimize auxiliary parameters rather than network inputs, limiting backpropagation to a small network subpart while preserving global output effects. Channel-wise scale and bias provide a speed–accuracy trade-off, with last-layer optimization causing only a small accuracy drop relative to full-network BRS.
- Background: BRS optimizes edits to click-conditioned inputs so the predicted mask becomes consistent with all user-provided labels.Its objective combines inertial and corrective energy, with λ regulating their trade-off.
- Background: 14?
- Background: Optimizing network inputs requires gradients through the whole model, creating substantial computational overhead.
- Feature backpropagating refinement: Channel-wise scale and bias affect outputs globally, avoiding the localized changes that make direct activation optimization ineffective.These parameters are invariant to image position and cannot produce the same degenerate localized solutions.
- Feature backpropagating refinement: f-BRS reparameterizes the network with auxiliary variables, allowing optimization through only a small part of the network while retaining equivalence at the initial parameters.The reparameterized function equals the original when the auxiliary parameters are fixed to their initial values.
- Feature backpropagating refinement: Applying f-BRS to the last several layers yields a natural speed–accuracy trade-off, with only a small accuracy drop compared to full-network BRS.
4. Zoom-In for interactive segmentation
Zoom-In crops the image around the inferred object to recover fine segmentation details while improving interactive segmentation quality and speed. The method uses an enlarged bounding box to preserve context and is applied by default after consistently improving results.
- Prior crop-based inference can preserve fine details for small objects but may degrade results when objects are too large for one crop.
- Zoom-In is introduced as an alternative technique intended to improve both segmentation quality and speed.
- Zoom-In is presented as an apparently new technique in interactive segmentation literature.
- After roughly three clicks, the method crops around the inferred mask’s bounding box, extending each side by 40% to retain context and boundary details.
- Experiments found consistent improvements with Zoom-In, motivating its default use across the paper’s experiments.
5. Experiments
Experiments evaluate f-BRS on four standard benchmarks using click-based interaction and show improved convergence, accuracy, and speed over feed-forward and BRS baselines.
- Evaluation setup: Experiments use GrabCut, Berkeley, DAVIS, and SBD with NoC@85 and NoC@90 measuring clicks needed to reach target IoU.The evaluation follows a standard click-generation protocol and uses DeepLabV3+ as the network architecture.
- Convergence analysis: Feed-forward networks can fail to converge, with additional clicks not improving masks for many images across architectures and datasets.Without BRS, some images remain inaccurately segmented even after 100 clicks.
- Convergence analysis: Backpropagating refinement reduces failure cases and improves convergence compared with networks without BRS.Both original BRS and f-BRS substantially reduce images that cannot be accurately segmented after extensive user input.
- Convergence analysis: Limiting consecutive clicks passed to the network is optimal at 4 clicks for RGB-BRS and 8 clicks for f-BRS-B on GrabCut and Berkeley.Both methods can adapt outputs to user input without explicitly passing every click to the network.
- Comparison with previous works: f-BRS requires fewer clicks than conventional algorithms, indicating accurate masks with less user effort.Across the evaluated benchmarks, the paper reports state-of-the-art accuracy and speed.
- Comparison with different backbones: Across three backbones, performance differs little; ResNet-34 achieves the best GrabCut quality despite being lighter than ResNet-101.The target training loss is poorly correlated with the NoC metric.
- Comparison with previous works: f-BRS runs an order of magnitude faster than RGB-BRS while adding only small overhead over a pure feed-forward model.The f-BRS-B variant is about 8 times faster than RGB-BRS with very close NoC results.
6. Conclusions
The paper concludes that f-BRS refines intermediate features using partial-network backpropagation and achieves state-of-the-art accuracy and speed across four benchmarks.
- Conclusions: f-BRS operates on intermediate network features and runs forward and backward passes through only a small part of the network.The method uses backpropagating refinement while reducing the computational scope of optimization.
- Conclusions: Experiments on four standard interactive segmentation benchmarks achieve state-of-the-art results in both accuracy and speed.The experiments also show better convergence than pure feed-forward approaches.
- Conclusions: BRS and f-BRS adapt network outputs to user input without explicitly passing clicks to the network.The conclusion connects this behavior to the analysis of first clicks and refinement convergence.
Appendix A. Analysis of the average IoU according to the number of clicks
The appendix examines how average IoU changes with click count and identifies a local-minimum failure mode for f-BRS-B on DAVIS and SBD.
- Average IoU versus clicks: f-BRS-B shows drops on DAVIS and SBD at 9 clicks because it can sometimes fall into a bad local minimum.The issue concerns the average-IoU curves as the number of clicks increases.
- Average IoU versus clicks: Increasing the BRS loss regularization coefficient λ can solve the local-minimum issue but worsens convergence at large click counts.The appendix therefore identifies a trade-off between avoiding bad minima and maintaining later convergence.
Appendix B. Measuring the limitation of fBRS
The appendix measures f-BRS’s accuracy ceiling and finds that it depends strongly on dataset difficulty and annotation characteristics.
- Accuracy limitation: The attainable accuracy limit of f-BRS is highly dependent on the dataset.The appendix estimates this limit using an oracle optimization procedure based on ground-truth masks.
- Accuracy limitation: DAVIS and SBD are harder than GrabCut and Berkeley for the method’s attainable accuracy.DAVIS contains complex pixel-precise masks, while SBD includes many rough or inaccurate annotations.
Appendix C. Full evaluation results for all our methods
Across GrabCut, Berkeley, SBD, and DAVIS, BRS consistently improves accuracy, while backbone effects are generally small but can become significant on SBD.
- NoC@85 and NoC@90 are reported across four datasets and three backbones for all BRS variations.These results are presented in Table 5.
- BRS consistently improves accuracy across the evaluated datasets and backbones.
- Backbone choice only slightly affects accuracy on GrabCut and Berkeley.
- ResNet-34 and ResNet-101 differ significantly on the SBD validation dataset.SBD has the closest distribution to the training data.
- DistMap-BRS generally achieves slightly worse NoC than RGB-BRS.
Appendix D. Additional interactive segmentation results
Additional examples illustrate both successful and unsuccessful behavior of f-BRS-B, including particularly difficult DAVIS cases that remain below the target IoU after many clicks.
- Figures 7 and 8 show good convergence of f-BRS-B with a ResNet-50 backbone on Berkeley images.
- Figure 9 shows bad Berkeley cases for f-BRS-B with a ResNet-50 backbone.
- In some of the worst DAVIS examples, the algorithm does not reach 85% IoU within 20 clicks.
Appendix E. Loss function ablation
The loss ablation uses normalized focal loss as an alternative to binary cross entropy, while the accompanying figures show convergence examples and challenging cases across datasets.
- Loss function ablation: Normalized focal loss is used as an alternative to binary cross entropy in the experiments.The ablation compares models trained with NFL and BCE loss functions.
- Loss function ablation: NFL focuses training on erroneous regions while retaining a total gradient equal to BCE's total gradient.
- Loss function ablation: The authors report that NFL training converges faster and achieves better accuracy than BCE training.
- Additional evaluation figures: Figure 6 compares average IoU as the number of clicks increases on GrabCut, Berkeley, DAVIS, and SBD.Its dashed horizontal line marks the theoretical average IoU limit for f-BRS-B.
- Additional evaluation figures: Figures 7 and 8 present good f-BRS-B convergence cases on Berkeley, while Figure 9 presents challenging Berkeley examples.
- Additional evaluation figures: Figure 10 presents some of the worst results on DAVIS.