PaperlayerView exact revision on arXiv

Source-linked AI summary

Cost-efficient Active Learning for Referring Image Segmentation and Grounding

Junbeom Hong, Seonghoon Yu, Hyung Rok Jung, Sundong Kim, Jeany Son

arXiv:2608.30621v2cs.CVcs.AI

TL;DR

Visual grounding annotation is costly because it requires discriminative referring expressions and region labels, yet existing active learning often assumes text is already available. The paper selects ambiguous raw-image samples using auxiliary region-text pairs and Referred Region Ambiguity, then accelerates expression labeling with a click-based interface. The framework outperforms several AL baselines on RIS and REC, while the interface achieves up to 1.6× faster labeling in a user study.

  • Problem

    Active learning for visual grounding lacks a realistic raw-image setting, even though annotation requires both referring expressions and corresponding regions.

  • Method

    The framework generates auxiliary region-text pairs, selects samples using Referred Region Ambiguity, and supports expression construction through region-aware word suggestions and clicks.

  • Results

    The framework consistently outperforms several AL baselines on RIS and REC, and referring-expression labeling is up to 1.6× faster than naïve labeling.

  • Takeaways & Limitations

    Ambiguous cross-region competition provides a practical signal for selecting informative visual-grounding samples while reducing human expression-labeling effort.

  • Takeaways & Limitations

    The interface incurs additional GPU memory and computation for multimodal-LLM word suggestions, which the paper leaves for future GPU-aware design.

Abstract

from arXiv · show

Collecting natural-language referring expressions along with region annotations, such as masks or boxes, is a major bottleneck in visual grounding (VG), as annotators must write descriptions that distinguish target regions from visually similar ones. We tackle this by formulating active learning (AL) for VG under the realistic setting where only raw images are available without accompanying text. Since ground-truth text is unavailable, sample selection must estimate which images contain ambiguous regions that would require discriminative referring expressions. To address this, we generate auxiliary region-text pairs using foundation models, and introduce Referred Region Ambiguity, a new acquisition function that measures whether the model's confidence collapses onto a single region or disperses across multiple candidates. It allows our method to prioritize images with strong cross-region competition, which are more informative due to their visual ambiguity. We also design a referring-expression annotation interface that helps annotators quickly focus on writing discriminative language with a few clicks. Experiments on RIS and REC benchmarks show that our AL framework consistently outperforms several AL baselines, while a user study shows up to 1.6X faster description labeling of ours.

1 Introduction

The paper frames active learning for visual grounding as raw-image selection, where both referring expressions and region annotations must be collected. It proposes ambiguity-aware selection and click-assisted expression labeling to improve informativeness and reduce annotation effort.

  • Active learning for visual grounding must select samples from raw images because unlabeled pools typically lack referring expressions and region labels.
  • High confidence across multiple candidate regions is informative in visual grounding because it exposes weak fine-grained discrimination.
  • Referred Region Ambiguity uses auxiliary region-text pairs to prioritize images where confidence is distributed across competing candidate regions.
  • The annotation interface lets users specify a target region and construct distinct expressions by sequentially selecting region-aware top-k word suggestions.
  • 1.6× faster labeling speeds were achieved for referring expressions than naïve labeling in a user study, while the framework outperformed several AL baselines on RIS and REC.

2 Related Work

Prior active-learning work largely targets vision-only tasks or vision-language settings with pre-existing text, while referring-expression annotation remains underexplored. This paper addresses raw-image visual grounding and jointly reduces sample-selection and expression-labeling costs.

  • Active learning research spans uncertainty, diversity, and hybrid criteria across classification, detection, segmentation, and VQA.
  • Vision-language active learning typically assumes pre-existing language inputs, unlike this work’s raw-image setting requiring target regions and referring expressions.
  • Prior cost-efficient annotation methods mainly reduce mask costs, leaving referring-expression annotation comparatively unexplored.
  • Visual grounding research includes weakly supervised, semi-supervised, zero-shot, and pseudo-labeling approaches for reducing supervision.

3 Active Learning for Visual Grounding

The framework selects informative raw images for visual grounding by generating auxiliary region-text pairs and measuring ambiguity across candidate regions before human annotation.

  • 3.1 Overall Framework: The active-learning setting selects samples directly from raw images, without pre-existing image-text pairs.Each round scores unlabeled images, annotates the highest-scoring subset within budget, and retrains the grounding model.
  • 3.2 Auxiliary Region-Text Pairs: Auxiliary region-text pairs provide multimodal inputs for estimating the informativeness of text-less images.Grounded-SAM extracts regions using COCO categories, while ViP-LLaVA generates captions for highlighted regions.
  • 3.3 Acquisition Function: Referred Region Ambiguity scores images by measuring whether confidence is distributed across multiple candidate regions for each auxiliary caption.High ambiguity indicates difficulty identifying the intended referent among visually or semantically similar regions.
  • 3.3 Acquisition Function: Normalized entropy increases when the model assigns similar confidence to multiple candidate regions instead of one dominant region.The image-level score averages normalized entropy across its auxiliary captions.
  • 3.3 Acquisition Function: Images with the highest informativeness scores are annotated with the cost-efficient labeling tool until the round budget is exhausted.The procedure repeats across active-learning rounds using the updated labeled set.

4 Cost-efficient Annotation Interface

The annotation interface reduces referring-expression labeling effort through click-driven region specification and sequential word selection, while modeling manual typing cost when suggestions are unsuitable.

  • Target-Region Specification: Annotators specify target regions through mask refinement for RIS or two-click bounding boxes for REC.RIS refinement merges pre-extracted SAM masks selected through clicks.
  • Referring-Expression Annotation: Annotators construct referring expressions by sequentially selecting region-aware top-k word suggestions or choosing a typing option.The process avoids drawing masks and typing complete sentences from scratch.
  • Cost Measurement: Referring-expression cost sums per-word selection costs, charging log2(k + 1) bits for selecting among k suggestions plus typing cost when needed.The typing cost uses conditional information content estimated by a frequency-based letter-level 3-gram model.

5 Experiments

Experiments evaluate the framework on RIS and REC under measured annotation-cost and acquired-pair budgets, separating acquisition quality from tool efficiency. The proposed acquisition and annotation interface consistently improve performance, while ablations and qualitative analyses support their effectiveness for selecting challenging, informative samples.

  • Experimental Setup: The framework is evaluated on RefCOCO, RefCOCO+, and RefCOCOg using mIoU for RIS and Prec@50 for REC.RIS results average per-sample IoU, while REC counts predictions correct when IoU exceeds 0.5.
  • Main Results: RIS experiments average mIoU across validation and test splits and compare annotation budgets from 2% to 30% against random acquisition and naïve labeling.The x-axis measures acquired-sample annotation cost divided by the dataset’s total naïve labeling cost.
  • Main Results: The proposed acquisition outperforms random acquisition under both efficient and naïve labeling, while the annotation interface performs strongly across all RIS budgets.These comparisons evaluate both selection quality and labeling efficiency.
  • Main Results: REC experiments show that the acquisition consistently beats random selection, while the text tool achieves higher performance than naïve typing at equal annotation cost.Results are averaged over validation and test splits for RefCOCO, RefCOCO+, and RefCOCOg at 10%, 20%, and 30% budgets.
  • Acquisition Comparison: Using acquired region-text pairs as the budget, Referred Region Ambiguity consistently outperforms random selection, pixel entropy, and coreset on RIS and REC.This comparison isolates acquisition effectiveness from annotation-tool cost.
  • User Study: A user study with 30 annotators compares the proposed interface with naïve polygon clicking and manual typing using time and quality metrics.The interface achieves faster annotation while maintaining high-quality mask specification and referring-expression labeling.
  • Ablation Study: Ablations show that replacing random selection with Referred Region Ambiguity helps especially at low budgets, while combining efficient mask and text tools yields the best results.The auxiliary generators’ moderate noise often does not change candidate priority because the pairs serve as acquisition proposals rather than final supervision.
  • Qualitative Analysis: The acquisition often selects images containing visually similar distractor regions, and their high feature similarity indicates that the method prioritizes challenging VG samples.The analysis measures similarity between the referred region and other regions within each selected image.

6 Conclusion

The paper presents active learning for visual grounding from raw-image pools, using auxiliary region-text samples, a tailored acquisition function, and an efficient annotation interface. It reports reduced manual effort and consistent outperformance of AL baselines on RIS and REC benchmarks.

  • Conclusion: The framework operates on purely raw-image pools, using auxiliary region-text samples as signals for active learning.It combines VG-specific sample informativeness estimation with an interface intended to simplify manual labeling.
  • Conclusion: The approach significantly reduces overall manual effort while consistently outperforming all AL baselines on RIS and REC benchmarks.The conclusion attributes the result to tailored acquisition and the proposed annotation interface.

7 Limitations

The annotation interface reduces human labor but adds GPU overhead that the current design does not explicitly account for. The authors leave GPU-aware interface design for future work.

  • 7 Limitations: The annotation tool reduces human labeling costs but introduces extra GPU memory and computation for multimodal word suggestions.The overhead arises during annotation-time generation of word-level suggestions.
  • 7 Limitations: The interface design does not explicitly consider the GPU costs incurred by the tool.The authors identify GPU-aware annotation-tool design as future work.
  • 7 Limitations: The authors characterize the additional GPU costs as marginal relative to the saved human labor.The interface runs on a single RTX 3090 and requires modest computation compared with the saved labor.

8 Ethical Consideration

The work emphasizes responsible human-in-the-loop annotation, including clear task communication, fair compensation, and verification of referring-expression quality and bias.

  • 8 Ethical Consideration: Human annotators should be clearly informed about the task and fairly compensated.The ethical consideration applies because the annotation tool relies on human annotators.
  • 8 Ethical Consideration: Human verification remains important for accurate and unbiased referring expressions.The interface reduces repetitive manual effort but does not remove the need for human review.

A.5 Analysis on GPU Cost

The framework incurs GPU costs for auxiliary-pair generation and annotation support, but these costs are compared against substantial human labor costs under a common dollar budget.

  • A.5 Analysis on GPU Cost: The framework adds GPU overhead through auxiliary-pair generation and annotation support.Grounded-SAM and ViP-LLaVA generate auxiliary pairs, while ViP-LLaVA and offline SAM support annotation.
  • A.5 Analysis on GPU Cost: 844 GPU hours are estimated for the labeling interface, including ViP-LLaVA text annotation and offline SAM mask extraction.The text interface accounts for 840 hours and mask pre-extraction for 4 hours.
  • A.5 Analysis on GPU Cost: 2,287 hours of human labor are estimated for naïve annotation of all 42,404 RefCOCO samples.The estimate combines 761 hours for polygon drawing with 1,526 hours for manually typing full sentences.
  • A.5 Analysis on GPU Cost: The additional GPU cost is described as minimal compared with the human labor cost required for large-scale manual annotation.The comparison is made under the same budget framework.

A.8 Analysis on Collected Expressions

The collected-expression analysis compares naïve typing with the proposed interface using language composition, diversity, ambiguity, quality, and referring accuracy, while also validating the text-cost model.

  • A.8 Analysis on Collected Expressions: The analysis measures expression attributes, spatial relations, object categories, semantic diversity, and ambiguity.Word categories are assigned with ChatGPT, diversity uses Vendi Score, and ambiguity is assessed through paired comparison.
  • A.8 Analysis on Collected Expressions: 90 annotators rate expression quality on a five-point scale, while referring accuracy is assessed by identifying which expression better refers to the target object.The additional study uses Amazon Mechanical Turk.
  • A.8 Analysis on Collected Expressions: Naïve typing produces higher diversity, but some longer expressions describe the overall image rather than the target region.The authors attribute the higher diversity to longer expressions containing more concepts.

B.4 Dataset

The section describes RefCOCO, RefCOCO+, and RefCOCOg, emphasizing their expression characteristics, collection procedures, and dataset sizes.

  • RefCOCO contains 142,210 expressions referring to 50,000 instances across 19,994 images.
  • RefCOCO+ prohibits positional terms, requiring expressions to rely primarily on attributes such as clothing color or object size.
  • RefCOCO+ includes 141,564 expressions for 49,856 instances across 19,992 images.
  • RefCOCOg expressions average 8.43 words, substantially longer than those in RefCOCO and RefCOCO+.
  • RefCOCOg was collected through Amazon Mechanical Turk and contains 95,010 expressions for 49,822 instances across 25,799 images.

C.1 Examples of User Study Results

The appendix provides qualitative evidence that the proposed annotation interface improves mask and expression labeling, while Referred Region Ambiguity selects complex, visually competitive scenes that support stronger RIS performance.

  • User Study Results: The interface produces more accurate masks than manual polygon drawing by using precise pre-extracted SAM masks for complex object boundaries.The mask tool also enables faster annotation times.
  • User Study Results: Contextually appropriate word suggestions help annotators create concise, accurate referring expressions, reflected in higher Sentence-BERT similarity than manual typing.The suggestions are generated step by step by ViP-LLaVA and relate to the target region.
  • Auxiliary Region-Text Pairs: Generated auxiliary region-text pairs generally localize objects accurately and provide clear, detailed referring expressions, despite minor noise in some pairs.The pairs are visualized with matching colors and their quality is quantified in Tab. 4.
  • Acquisition Qualitative Results: On the curated ambiguous subset, RIS models trained with Referred Region Ambiguity better distinguish targets from visually similar distractors than models trained with other acquisitions.The comparison includes random, coreset, and pixel-entropy baselines.
  • Acquisition Qualitative Results: Referred Region Ambiguity prioritizes complex scenes containing many objects, producing more informative selections than pixel-entropy and coreset.Additional selected samples are shown in Fig. 20 with matching colors for paired ground-truth region-text annotations.
  • Informativeness Score Analysis: Unlike pixel-entropy, which averages entropy over pixels and can assign low informativeness to scenes with similar objects, the region-level acquisition yields high scores for these complex scenes.The qualitative comparison illustrates how region-level scoring captures scene complexity.
Loading 2608.30621v2…