PaperlayerView exact revision on arXiv

Source-linked AI summary

DropClick: Semi-Automated One-Click Segmentation for Agricultural Robotic Data

Patrick Zimmer, Michael Halstead, Chris McCool

arXiv:2609.03680v1cs.CVcs.RO

TL;DR

Segmentation annotation is costly, and click-based methods commonly require input for every object. DropClick predicts masks for clicked and unclicked objects from partial clicks, then supplies pseudo-labels for downstream training. Across the two agricultural datasets, performance remains high with half the clicks missing, including downstream Mask2Former results close to those with all clicks.

  • Problem

    Segmentation datasets require costly, laborious labels, and existing click-based approaches commonly require input for every object.

    Source: §I · p. 1

  • Method

    DropClick takes optional object clicks and predicts masks for both clicked and non-clicked objects, whose outputs can serve as pseudo-labels for downstream segmentation training.

    Source: §I · p. 2, §III · p. 3

  • Results

    With 50% of clicks missing, mIoU decreases by only 1.1 points on SB20 and 1.3 on BUP20; downstream SB20 Mask2Former reaches 70.1 AP50 versus 70.7 with all clicks, while BUP20 reaches 77.0 for both.

    Source: A. One-click segmentation · p. 6, §VI · p. 8

  • Takeaways & Limitations

    The results support using DropClick to generate agricultural segmentation pseudo-labels while reducing the need for user clicks on every object.

    Source: §I · p. 2, §VI · p. 8

Abstract

from arXiv · show

Labelling vision datasets, especially for segmentation tasks, is a laborious and costly process that stymies novel developments in agricultural robotics. In this paper, we present DropClick, a click-guided segmentation tool that simplifies the annotation process. Our system utilises single-click inputs on objects to generate pseudo-labels, which can replace manual annotations. DropClick stands out as it is a semi-automated approach and does not require a click for every object in the scene. It can therefore further reduce the required amount of user input drastically. We evaluate our method on two challenging agricultural robotic datasets, SB20 and BUP20 for plant and fruit segmentation, respectively. DropClick is first trained on a small subset of just 5 images from the original training data. This DropClick model can then be deployed as a one-click segmentation system and achieves comparable or higher performance than other one-click methods achieving an mIoU of 70.0 and 72.6 points, for SB20 and BUP20 respectively. DropClick then excels at maintaining high performance when clicks are not given (e.g. dropped); when 50% of the clicks are missing it still maintains an mIoU of 68.9 and 71.3 points, for SB20 and BUP20 respectively. We validate DropClick as a pseudo-labelling approach by taking its outputs to train a Mask2Former instance-based segmentation model in a semi-supervised manner. In this process, partially removing user input from DropClick yields similar high performance when compared to providing all clicks, at 70.1 vs 70.7 points AP50 for SB20 and no difference for BUP20 at 77.0 for both models; at the same time saving 46.3% of total input for SB20 and 31.9% for BUP20.

I. INTRODUCTION

Segmentation labels are costly to produce for agricultural robotics, while existing click-based approaches generally require input for every object. DropClick predicts masks for clicked and unclicked objects, and is evaluated on two agricultural datasets using five hand-annotated training images.

I. INTRODUCTION

  • Existing click-based segmentation typically requires user input for every object, although annotators would ideally click only difficult or occluded objects.Agricultural vision systems rely on labeled data, and annotation is expensive and laborious.Source: §I · p. 1
  • DropClick uses optional object clicks to predict masks for both clicked and non-clicked objects in a single network pass.The authors propose its outputs as pseudo-labels to reduce manual labelling effort.Source: §I · p. 2
  • Evaluation covers SB20 sugar beet crops and BUP20 sweet peppers, with DropClick trained on only five hand-annotated images.The authors also use DropClick-generated masks to train Mask2Former semi-supervised on the remaining training images.Source: §I · p. 2
  • The authors present DropClick as a competitive one-click tool that reduces pseudo-labeling input across two agricultural datasets.They frame it as a semi-automated alternative to requiring a click for every object.Source: §I · p. 2

II. RELATED WORK

The related work connects agricultural robotics’ need for segmentation labels with instance-segmentation methods and their annotation demands. It identifies Mask2Former as a transformer-based foundation for the proposed system.

II. RELATED WORK

  • Manual segmentation annotation is slow and costly, motivating guidance networks that predict labels from clicks, boxes, or text.Clicks are described as a simple form of guidance; interactive systems commonly refine masks iteratively.Source: §II · p. 2, A. Transformers for instance… · p. 2

A. Transformers for instance segmentation

  • Instance segmentation can provide object outlines useful for agricultural robotics, including shape estimates for locating object centers.The section cites plant and fruit tracking as an application.Source: A. Transformers for instance… · p. 2
  • Mask R-CNN relies on NMS postprocessing, whereas DETR removes that inference step through learned queries and Hungarian matching; Mask2Former adapts this approach to segmentation.The paper identifies Mask2Former as a strong foundation for follow-up work.Source: A. Transformers for instance… · p. 2

B. Click-guided segmentation

Click-guided segmentation ranges from iterative interactive refinement to transformer methods that process multiple objects together. The section positions DropClick against the limitation that existing approaches generally require input for each object, and describes its semi-automated alternative.

B. Click-guided segmentation

  • Interactive segmentation commonly refines masks iteratively with positive object clicks or negative background clicks, using clickmaps or previous interaction outputs.Transformer methods such as SAM and SAM2 instead use positional encodings and prompt embeddings, while DynaMITe passes clicks as individual queries.Source: B. Click-guided segmentation · p. 2
  • DynaMITe and panoptic one-click methods process multiple objects in a scene together, unlike approaches that handle objects separately.The section notes the joint approach can improve computational efficiency and leverage relationships between objects.Source: B. Click-guided segmentation · p. 3
  • DropClick addresses the per-object click requirement by producing predictions for objects both with and without user-supplied clicks.The authors describe the approach as inspired by DynaMITe and Mask2Former.Source: B. Click-guided segmentation · p. 3

C. One-click segmentation baselines

  • The authors compare general one-click performance with SAM2 and Panoptic One-Click, while noting prior approaches do not generally automate segmentation of unclicked objects.They state that earlier work had not further examined whether missing-click recovery could semi-automate the task.Source: C. One-click segmentation baselines · p. 3, B. Click-guided segmentation · p. 3
  • Figure 2 shows initial proposals, optional clicks to refine faulty predictions, and final masks for clicked and non-clicked objects.False-positive predictions can remain in the accepted result and are typically removed manually after inference.Source: Figure 2 · p. 3

III. PROPOSED APPROACH

DropClick combines click-guided and automatic instance segmentation to produce masks for clicked and non-clicked objects. Users can accept proposals or refine faulty predictions with further clicks; missing and overlapping proposals are especially important targets.

III. PROPOSED APPROACH

  • DropClick combines one-click-guided and automatic instance segmentation to predict masks for both clicked and non-clicked objects.Users receive mask proposals that they can accept or refine with further clicks.Source: §III · p. 3
  • Clicks are most important for missing or overlapping proposals, while the number and order of clicks are arbitrary.The workflow includes additional clicks for false positives, which would generally be deleted after the iterative process.Source: §III · pp. 3–4
  • For non-clicked objects, DropClick uses 100 learnt foreground queries and Hungarian matching to produce instance-based masks alongside click-query masks.The system produces masks for all objects regardless of click status.Source: Figure 3 · p. 4

A. DropClick

DropClick builds on a transformer-based interactive segmentation approach, adapting it to single clicks and adding instance-based segmentation for objects without clicks. At inference, it combines masks from click queries and learnt foreground queries to cover both clicked and non-clicked objects.

A. DropClick

  • DropClick adapts DynaMITe by setting temporal click encoding to zero and training with one click per object instead of multiple clicks.DynaMITe uses click queries and learnt background queries in a transformer-based multi-object interactive segmentation architecture.Source: A. DropClick · p. 4
  • DropClick adds instance-based segmentation of non-clicked objects using 100 learnt foreground queries and a classification head.The queries are matched to non-clicked ground-truth objects during training, and classification loss is applied only to the learnt foreground queries.Source: A. DropClick · p. 4
  • At inference, DropClick combines masks from classified foreground queries for non-clicked objects with click-query masks for clicked objects.For each group, pixel assignments are made by applying argmax with background-query outputs.Source: A. DropClick · p. 5

IV. EXPERIMENTAL SETUP

The setup evaluates DropClick on two agricultural datasets and trains it using five hand-selected images per dataset. It also specifies how clicks are derived, randomized, and added in missing-click evaluations.

A. Datasets

  • SB20 contains sugar beet crops in arable farming and BUP20 sweet peppers in a glasshouse, with 71/124 training, 37/63 validation, and 35/93 evaluation images, respectively.Both datasets include instance-segmentation labels and challenges such as high intra-class variability and heavy occlusions.Source: A. Datasets · p. 5
  • DropClick is trained on five hand-selected training images per dataset; downstream Mask2Former training uses generated pseudo-labels for the remaining images without a validation set.The stated rationale for omitting validation data is that users in real-world scenarios may lack manually annotated validation data.Source: A. Datasets · p. 5

B. Click inputs

  • SB20 clicks use available stem-location labels, while BUP20 clicks use mask-derived centers of mass, with binary erosion when a center lies outside the object.BUP20 centers are derived from the original mask labels.Source: B. Click inputs · p. 5
  • Training randomizes clicks each epoch with a 2D Gaussian centered on calculated click locations, with greater variation allowed for larger objects.The randomization radius is R = max(a, b)/5 and its standard deviation is σ = R/3, where a and b are bounding-box side lengths.Source: B. Click inputs · p. 5
  • Missing-click evaluation adds clicks first for false negatives in ascending IoU order, then for overlapping predictions in descending IoU order, and finally for remaining objects randomly.A prediction is treated as a false negative when its IoU is below 0.4.Source: B. Click inputs · p. 5

C. Implementation and metrics

The implementation section describes training settings and evaluation metrics for DropClick and the one-click baselines. The named baselines include Panoptic One-Click Segmentation and SAM2.

C. Implementation and metrics

  • The implementation section states that each experiment specifies model training and metrics for evaluating click-based segmentation with partly missing user input.Source: C. Implementation and metrics · p. 5

1) One-Click Segmentation Baselines:

  • The Panoptic One-Click Segmentation baseline follows its original settings, with click randomization and the dataloader aligned to DropClick; SAM2 is fine-tuned for 200 epochs.Performance is measured using mean object IoU (mIoU) on evaluation sets.Source: 1) One-Click Segmentation Baselines · p. 5

2) DropClick:

DropClick is trained to predict both clicked and unclicked objects by varying click input during training. The paper also describes its loss setup, evaluation of missing clicks, and use of generated pseudo-labels to train Mask2Former.

2) DropClick:

  • Training varies click input: 25% of iterations drop no clicks, 25% drop all clicks, and 50% drop a random number between 1 and Nobj −1.This exposes the network to both clicked and non-clicked objects.Source: 2) DropClick · p. 5
  • Mask losses are computed separately for clicked and learnt queries with different weights, while classification loss applies only to learnt queries.Clicked-query BCE and dice weights are 5 each; learnt-query weights are 10 and 25, and classification weight is 10.Source: 2) DropClick · p. 5
  • Missing-click evaluation reports mIoU and false-negative and false-positive counts, with Nmiss = ⌈rmiss ∗ Nobj⌉.The rounded-up count can make the realized missing-click ratio higher than the chosen ratio, especially in images with few objects; the authors say this is not the case in their data.Source: 2) DropClick · p. 6

3) Semi-supervised instance segmentation:

  • DropClick-generated pseudo-labels train Mask2Former instance segmentation semi-supervisedly, with performance measured as AP50.The described setup uses a Swin-Large backbone and 50 training epochs.Source: 3) Semi-supervised instance segmentation · p. 6

V. RESULTS

DropClick provides strong one-click segmentation results on SB20 and BUP20, including when about half the object clicks are missing. The results compare it with SAM2 and Panoptic One-Click and illustrate predictions on both datasets.

V. RESULTS

  • DropClick experiments compare mask quality against SAM2 and Panoptic One-Click Segmentation, vary the number of missing clicks to assess performance, and test pseudo-labels for semi-supervised instance segmentation.Source: §V · p. 6

A. One-click segmentation

  • DropClick exceeds SAM2 by 2.6 mIoU points on SB20 and 1.5 on BUP20; against Panoptic One-Click it is 1.9 points lower on SB20 but 14.0 higher on BUP20.Table I covers evaluation sets, with models trained on five-image subsets and DropClick evaluated with all, ~50% missing, or no clicks.Source: Table I · p. 6, A. One-click segmentation · p. 6
  • Figure 4 marks true-positive masks for unclicked objects in orange, clicked objects in green, and false positives in magenta.Each false positive requires one additional removal click after inference.Source: Figure 4 · p. 6
  • With about 50% of clicks missing, DropClick loses 1.1 mIoU points on SB20 and 1.3 on BUP20, and still outperforms SAM2.With all input removed, performance decreases by 8.2 points on SB20 and 14.3 on BUP20.Source: A. One-click segmentation · p. 6, Figure 5 · p. 7

B. Removing Input Clicks

DropClick maintains high mask quality without the full set of clicks, and the authors use about 50% missing clicks as a reliable setting across both datasets. Pseudo-label generation at that setting saves user input, although false positives add removal clicks.

B. Removing Input Clicks

  • mIoU and false negatives reach a saturation point with 70% of clicks still missing on SB20 and 50% on BUP20, motivating 50% as the shared setting.The authors report that all metrics improve as the percentage of missing clicks decreases.Source: B. Removing Input Clicks · p. 7
  • At 50% missing clicks, DropClick records 1.5 false positives per image on SB20 and 5.9 per image on BUP20, while still yielding high-quality masks for clicked and non-clicked objects.Source: B. Removing Input Clicks · p. 7
  • 46.3% of total input is saved on SB20 and 31.9% on BUP20, with pseudo-label mIoU of 69.0 and 70.0, respectively.The click savings include both missing object clicks and clicks to remove false positives; pseudo-label quality is described as comparable to the evaluation sets.Source: B. Removing Input Clicks · p. 7, Table II · p. 7

C. Semi-supervised Instance Segmentation

DropClick pseudo-labels support semi-supervised Mask2Former instance segmentation with performance close to full-click pseudo-label training. Partially removing clicks also saves substantial input on both datasets.

C. Semi-supervised Instance Segmentation

VI. CONCLUSION

  • 46.3% of total input is saved on SB20 and 31.9% on BUP20 while maintaining high performance with partially removed clicks.The conclusion reports that the system was applied to instance segmentation of plants and sweet peppers.Source: §VI · p. 8

Related papers

Loading 2609.03680v1…