PaperlayerView exact revision on arXiv

Source-linked AI summary

Extreme clicking for efficient object annotation

Dim P. Papadopoulos, Jasper R. R. Uijlings, Frank Keller, Vittorio Ferrari

arXiv:1708.02750v1cs.CV

TL;DR

Manual bounding-box annotation is time-consuming because it requires manipulating imaginary corners and adjusting a tight rectangle. The paper replaces this with four physical extreme-point clicks, showing faster annotation with comparable box and detector quality, while the points also support better segmentation.

  • Problem

    High-quality manual bounding-box annotation is time-consuming because annotators must locate and adjust imaginary box corners.

  • Method

    The paper uses extreme clicking, asking annotators to click the object’s top, bottom, left-most, and right-most points, and incorporates the points into GrabCut.

  • Results

    7s per box is achieved instead of 34.5s, with comparable box quality and detector performance, while GrabCut and derived semantic segmentations also improve.

  • Takeaways & Limitations

    Extreme clicking reduces annotation cost without compromising box quality or detector accuracy and additionally supplies boundary points useful for segmentation.

Abstract

from arXiv · show

Manually annotating object bounding boxes is central to building computer vision datasets, and it is very time consuming (annotating ILSVRC [53] took 35s for one high-quality box [62]). It involves clicking on imaginary corners of a tight box around the object. This is difficult as these corners are often outside the actual object and several adjustments are required to obtain a tight box. We propose extreme clicking instead: we ask the annotator to click on four physical points on the object: the top, bottom, left- and right-most points. This task is more natural and these points are easy to find. We crowd-source extreme point annotations for PASCAL VOC 2007 and 2012 and show that (1) annotation time is only 7s per box, 5x faster than the traditional way of drawing boxes [62]; (2) the quality of the boxes is as good as the original ground-truth drawn the traditional way; (3) detectors trained on our annotations are as accurate as those trained on the original ground-truth. Moreover, our extreme clicking strategy not only yields box coordinates, but also four accurate boundary points. We show (4) how to incorporate them into GrabCut to obtain more accurate segmentations than those delivered when initializing it from bounding boxes; (5) semantic segmentations models trained on these segmentations outperform those trained on segmentations derived from bounding boxes.

1. Introduction

Traditional box annotation is slow because annotators must manipulate imaginary corners, switch tasks, and adjust box sides. Extreme clicking replaces this with four physical object-boundary clicks and preserves annotation and downstream detection quality while enabling improved segmentation.

  • 35s per box is required by an efficient traditional crowdsourcing protocol for high-quality bounding-box annotation.
  • Traditional drawing combines corner clicking, dragging, side adjustment, task switching, and mental imagery.The imagined rectangle and corners are often outside the object, requiring annotators to locate extreme points and adjust coordinates.
  • Extreme clicking asks annotators to click the top, bottom, left-most, and right-most physical points of the object.The scheme removes rectangle construction, separate box adjustment, and a submit step.
  • Extreme clicking also provides four boundary points that can initialize GrabCut for more accurate segmentation than bounding-box initialization.The points improve object-appearance initialization and the choice of pixels clamped as object or background.
  • 7s per box is achieved with extreme clicking, while boxes match traditional ground truth in quality and detectors trained on them perform as well as those trained on original ground truth.

2. Related work

Prior work seeks faster alternatives to manual bounding-box annotation through improved crowdsourcing, weak supervision, video, auxiliary signals, and interactive segmentation. High-quality traditional box annotation remains a reference point for comparison.

  • 34.5s is the reference time for a high-quality box-drawing protocol, combining 25.5s drawing and 9.0s verification.Including rejected boxes and redrawing raises the median to 55s, while average-time estimates reach 117s.
  • The reference protocol was designed for high-quality boxes with little Amazon Mechanical Turk annotation time and was used for ILSVRC.Faster reported timings targeted lower-quality boxes, so the paper compares against the high-quality reference.
  • Video-based localization, weakly supervised localization, and alternative signals such as eye tracking, text, and movie scripts have been explored to reduce reliance on manual boxes.Video-trained detectors remain weaker than detectors trained on manually annotated still images because of domain adaptation.
  • Interactive segmentation methods ask users for bounding boxes, scribbles, or clicks and commonly build on GrabCut.

3. Collecting extreme clicks

The crowd-sourcing protocol combines simple four-point instructions with qualification, feedback, retryable testing, batching, and embedded quality control. These stages are designed to admit qualified annotators and maintain annotation accuracy efficiently.

  • 3.1. Instructions: Annotators are instructed to click the visible object’s top, bottom, left-most, and right-most points in any order.The protocol suggests 10s for all four clicks as an upper bound estimated from a pilot study.
  • 3.1. Instructions: The instructions are simpler than those needed to draw or verify a traditional bounding box.Traditional verification requires imagining a perfect box and mentally comparing it with the displayed box.
  • 3.2. Annotator training: Annotators complete a qualification test with detailed feedback before entering the annotation stage, and failed tests can be repeated.The test is intended to improve crowdsourcing quality and filter inattentive annotators and spammers.
  • 3.2. Annotator training: The qualification test validates clicks against accepted regions derived from ground-truth segmentation masks with a 10-pixel tolerance.Accepted regions include extreme mask pixels plus nearby image pixels inside and outside the mask.
  • 3.2. Annotator training: Feedback displays each annotator’s four clicks alongside the accepted area for every click.
  • 3.3. Annotating images: Annotation proceeds in batches of 10 consecutive images sharing one target class, reducing repeated class-name reading and task switching.
  • 3.3. Annotating images: A hidden evaluation image acts as a golden question, blocking submission when an annotator fails its accepted-area check.

4. Object segmentation from extreme clicks

The paper uses extreme boundary clicks to initialize GrabCut with a more informative object surface and object-label constraints than a bounding box alone.

  • Extreme clicking supplies four boundary points that can improve segmentation-mask creation from bounding boxes.
  • The energy combines unary appearance likelihoods with pairwise smoothness penalties over neighboring pixel labels.The paper uses edge-detector responses for its pairwise potential and alternates graph-cut labeling with GMM updates.
  • For each consecutive pair of extreme clicks, the method finds a likely object-boundary path using a strong edge detector.The selected path maximizes its minimum edge response among connecting paths.
  • The enclosed surface initializes GrabCut’s object appearance model, while its morphological skeleton is clamped to the object label.These steps provide both appearance-model initialization and pixels that are very likely to be object.
  • GrabCut models object and background appearance with separate five-component full-covariance RGB Gaussian mixture models.
  • Using extreme-click appearance initialization produces visibly better object-pixel posterior predictions than initializing the object model from the box interior.Both comparisons use a background appearance model formed from an area outside the initial box.

5. Extreme Clicking Results

Extreme clicking matches traditional box quality and detector performance while reducing annotation time, and it compares favorably with faster alternative protocols. Experiments also examine dataset scale, evaluation measures, crowd quality controls, and annotation cost.

  • Experimental setup: The study collects one instance per class when present across 5011 VOC 2007 trainval images and 5717 VOC 2012 training images.Both datasets contain 20 object categories, enabling direct comparisons with other methods.
  • Box quality: Extreme-click boxes match PASCAL ground-truth quality within the agreement level of expert annotators.On VOC 2007, GT Boxes and GT SegBoxes agree at 88% mIoU; 93% of GT Boxes exceed IoU 0.7 with their corresponding GT SegBox.
  • Detector performance: Fast-RCNN performance is identical when trained on extreme-click boxes or PASCAL GT Boxes across both datasets and both base networks.The experiments use VOC 2007 and 2012 with AlexNet and VGG16.
  • Efficiency: 7.0s instead of 34.5s makes extreme clicking 5× cheaper than the reference box-drawing protocol.The comparison uses 34.5s as the time for one high-quality traditional box.
  • Alternative protocols: Extreme clicking produces 97% correct boxes at IoU>0.5, compared with 81% for human verification, while verification is 1.6× faster.On VOC 2007, detector performance is also 6%-8% mAP higher for extreme clicking.
  • Alternative protocols: Extreme clicking reaches 66% mAP with VGG16, compared with 35% mAP for the cited weakly supervised method.The weakly supervised approach requires very little human supervision time.
  • Alternative protocols: 7s per box and 91%-94% IoU>0.7 boxes outperform the cited 12.3s box-drawing protocol, which achieves 71% IoU>0.7 boxes.The paper characterizes this comparison as approximate because the alternative measurements come from an unspecified ILSVRC 2014 subset.
  • Additional analysis: Qualification testing raises annotation quality from 75.4% to 85.7% mIoU, while quality control further raises it to 87.1% mIoU.These experiments use 200 VOC 2007 trainval images.

6. Results on Object Segmentation

Extreme clicking supplies boundary points that improve GrabCut segmentations over bounding-box initialization and produces training segmentations that approach full supervision.

  • The experiments used VOC 2007 and VOC 2012, with mIoU as the segmentation metric and VOC 2012 evaluation on 1,449 validation images.VOC 2007 included 422 trainval images with segmentation masks; VOC 2012 training used 5,623 images.
  • 74.4% mIoU on VOC 2007 and 71.0% mIoU on VOC 2012 were obtained with the optimized bounding-box GrabCut baseline.The model uses customized appearance initialization, clamping, and edge-based pairwise potentials.
  • 78.1% mIoU on VOC 2007 and 72.7% on VOC 2012 resulted when extreme boundary points initialized appearance models and object clamping.These results improved over the strong bounding-box baseline by 3.7% and 1.7%, respectively.
  • 5.5% error was achieved on GrabCut versus 8% from bounding boxes when boundary points were additionally used.The simulated extreme clicks came from ground-truth segmentation-mask extreme points.
  • 58.4% mIoU was achieved by models trained on extreme-click segmentations, compared with 55.8% from GT-box segmentations and 59.9% with full supervision.Extreme-click training improved over GT-box training by +2.6% mIoU and was -1.5% below full supervision using the same total number of training samples.

7. Conclusions

Extreme clicking replaces imaginary box-corner placement with four physical object-boundary clicks, preserving box quality while reducing annotation time and supporting improved segmentation.

  • 7s per annotation matched traditional box quality, while traditional drawing required 34.5s for the same quality.The paper reports a 5× reduction in annotation cost without compromising box quality.
  • Extreme clicking also provides accurate object-boundary points, which improve GrabCut segmentations over bounding-box initialization.Semantic segmentation models trained on these segmentations performed close to models trained with manually drawn segmentations using the same number of samples.

A. Qualitative examples of extreme clicking

Extreme-click boxes usually closely matched PASCAL ground-truth boxes, while disagreements concentrated in ambiguous or difficult object configurations.

  • 94% of extreme-click boxes had IoU>0.7 with the corresponding PASCAL ground-truth box.Only 6% of objects had annotations with IoU<0.7.
  • 62% of almost-no-overlap cases were correct extreme-click annotations for objects absent from the PASCAL ground truth.Other cases involved similar classes, wrong classes, or spatial annotation errors.
  • 21% of low-overlap cases favored extreme clicks, 23% favored PASCAL ground truth, and 56% were undecidable.Difficult cases commonly involved partial occlusion, thin parts, dark images, or small objects.
Loading 1708.02750v1…