PaperlayerView exact revision on arXiv

Source-linked AI summary

GRES: Generalized Referring Expression Segmentation

Chang Liu, Henghui Ding, Xudong Jiang

arXiv:2306.00968v1cs.CV

TL;DR

Classic RES datasets and methods mainly support one target per expression and omit no-target cases, limiting the task's practical scope. The paper introduces GRES, the gRefCOCO dataset, and the region-based ReLA baseline, which achieves state-of-the-art performance on both GRES and classic RES. The authors identify complex relationship modeling and difficult no-target cases as important challenges.

  • Problem

    Classic RES commonly supports only single-target expressions and does not consider multi-target or no-target expressions.

  • Method

    The paper defines GRES, constructs gRefCOCO with multi-target and no-target samples, and proposes ReLA to model region-region and region-language relationships.

  • Results

    ReLA achieves new state-of-the-art performance on both the newly proposed GRES and classic RES tasks.

  • Takeaways & Limitations

    GRES broadens referring segmentation to arbitrary numbers of targets and no-target expressions, increasing its supported application scope.

  • Takeaways & Limitations

    Failure cases show that ReLA remains challenged by deceptive relationships and fine-grained spatial details, while no-target identification is identified as a future focus.

Abstract

from arXiv · show

Referring Expression Segmentation (RES) aims to generate a segmentation mask for the object described by a given language expression. Existing classic RES datasets and methods commonly support single-target expressions only, i.e., one expression refers to one target object. Multi-target and no-target expressions are not considered. This limits the usage of RES in practice. In this paper, we introduce a new benchmark called Generalized Referring Expression Segmentation (GRES), which extends the classic RES to allow expressions to refer to an arbitrary number of target objects. Towards this, we construct the first large-scale GRES dataset called gRefCOCO that contains multi-target, no-target, and single-target expressions. GRES and gRefCOCO are designed to be well-compatible with RES, facilitating extensive experiments to study the performance gap of the existing RES methods on the GRES task. In the experimental study, we find that one of the big challenges of GRES is complex relationship modeling. Based on this, we propose a region-based GRES baseline ReLA that adaptively divides the image into regions with sub-instance clues, and explicitly models the region-region and region-language dependencies. The proposed approach ReLA achieves new state-of-the-art performance on the both newly proposed GRES and classic RES tasks. The proposed gRefCOCO dataset and method are available at https://henghuiding.github.io/GRES.

1. Introduction

Classic RES is constrained to single-target expressions and does not handle absent targets, motivating GRES, gRefCOCO, and a relationship-aware baseline ReLA. The paper reports that ReLA achieves state-of-the-art performance on both GRES and classic RES.

  • Limitations of classic RES: Classic RES excludes no-target expressions and usually lacks multi-target samples, limiting its behavior and requiring separate searches for multiple objects.Existing methods trained on classic datasets do not generalize well to these scenarios.
  • New benchmark and dataset: GRES extends RES to expressions indicating any number of target objects, including multi-target and no-target cases.GRES retains image-expression inputs while allowing masks covering multiple targets or an all-negative mask for no-target expressions.
  • New benchmark and dataset: gRefCOCO is a large-scale dataset that complements RefCOCO with multi-target and no-target samples for realistic referring segmentation research.The dataset is designed to support research on the expanded GRES setting.
  • A baseline method: ReLA is a baseline that models complex relationships among objects, addressing the stronger need for region interactions in multi-target expressions.The method is motivated by the long-range region-region dependencies introduced by GRES.
  • Experimental study: ReLA achieves new state-of-the-art performance on both the newly proposed GRES and classic RES tasks.The paper also reports extensive experiments comparing ReLA with existing RES methods on GRES.

2. Related Works

Classic referring datasets and methods largely follow a one-expression, one-instance convention, while newer datasets do not fully support the intended GRES setting. RES methods are commonly organized into one-stage and two-stage approaches.

  • Related referring tasks and datasets: ReferIt, RefCOCO, and RefCOCOg contain single-target expressions, establishing a de-facto one-expression, one-instance rule for RES.The original RES definition did not limit the number of target instances, but common datasets adopted this convention.
  • Related referring tasks and datasets: PhraseCut includes multi-target expressions only as a fallback and uses templates, unlike GRES's intentional multi-target free-language setting.This limits its suitability as a direct benchmark for GRES.
  • Generalized RES: GRES broadens referring segmentation beyond classic single-instance datasets by supporting multi-target and no-target expressions.Figure 2 highlights additional applications enabled by these expression types.
  • Referring segmentation methods: RES methods are broadly divided into one-stage networks that predict masks from fused features and two-stage networks that select targets from instance proposals.Most RES methods are one-stage, while two-stage methods first generate proposals and then select target instances.

3. Task Setting and Dataset

GRES extends referring expression segmentation to expressions referring to any number of objects, including multiple targets and no target. The gRefCOCO dataset supports this setting with diverse expressions and target configurations.

  • 3.1. GRES Settings: GRES predicts a mask covering all targets and an indicator for no-target expressions, for which the mask should be entirely negative.
  • 3.2. gRefCOCO: A Large-scale GRES Dataset: GRES supports practical expressions such as “all people,” “two players on left,” “foreground,” and “kids” for selecting multiple objects or open-vocabulary regions.
  • 3.1. GRES Settings: Evaluation adds generalized IoU and separate no-target performance to the conventional cIoU and Precision@X metrics.GRES does not require methods to differentiate individual instances, allowing popular one-stage RES methods to participate.
  • 3.2. gRefCOCO: A Large-scale GRES Dataset: gRefCOCO contains 278,232 expressions across 19,994 images, including 80,022 multi-target and 32,202 no-target expressions.The dataset refers to 60,287 distinct instances and includes masks and bounding boxes for all target instances.
  • 3.2. gRefCOCO: A Large-scale GRES Dataset: Multi-target expressions are designed around selected instance groups and require counting, compound-language understanding, attribute assignment, and complex relationship modeling.Examples include “A and B,” “A except B,” and expressions combining shared or distinct attributes.
  • 3.2. gRefCOCO: A Large-scale GRES Dataset: No-target annotation rules exclude expressions irrelevant to the image and permit deceptive expressions from the same RefCOCO split, improving diversity while maintaining reasonable difficulty.

4. The Proposed Method for GRES

ReLA addresses GRES’s complex interaction requirements by dynamically extracting image regions and modeling region-region and region-language dependencies. It predicts regional masks, target probabilities, and no-target judgments before fusing the regional masks.

  • 4.1. Architecture Overview: ReLA dynamically divides image features into P^2 regions and explicitly models interactions among image regions and expression words.The regions are dynamically found rather than hard-split, supporting flexible spatial representations.
  • 4.1. Architecture Overview: The architecture encodes the image and expression as Fi and Ft, decodes mask features Fm, and produces region masks Mr, probabilities xr, and no-target score E.The output mask is obtained by weighted fusion of the regional masks.
  • 4.1. Architecture Overview: ReLA supervises the predicted mask with the target mask, the region probability map with a downsampled minimap, and the no-target score with the no-target label.During inference, a positive no-target prediction sets the output mask to empty.
  • 4.2. ReLAtionship Modeling: RLA combines region self-attention with region-language cross-attention to model region-region and word-region dependencies.The resulting interaction-aware, language-aware, and image-region features are fused by an MLP.
  • 4.2. ReLAtionship Modeling: RIA uses learnable region queries and image attention maps to collect flexible region features from relevant spatial positions.A single instance may occupy multiple regions, enabling sub-instance representations such as a person’s head and upper body.

5. Experiments and Discussion

Experiments evaluate GRES metrics, dataset necessity, design choices, comparisons with RES methods, and qualitative strengths and failures. Results show that explicit relationship modeling improves performance, while no-target identification remains difficult.

  • Evaluation Metrics: GRES evaluates segmentation with cIoU and Pr@X, while N-acc. and T-acc. measure no-target identification and targeting generalization.Pr@X excludes no-target samples, and its threshold starts at 0.7 because multi-target samples have larger foreground areas.
  • Dataset Necessity: Models trained on RefCOCO fail to generalize reliably to multi-target and no-target expressions, whereas gRefCOCO enables handling arbitrary numbers of objects.RefCOCO-trained models find only one of two explicitly described targets and produce meaningless masks for no-target expressions.
  • Ablation Study: 5.59% gIoU separates average-pooled region features from rigid patches, demonstrating the importance of preserving global context in RIA.The ablation compares region-construction choices within the ReLA framework.
  • Ablation Study: 1.54% gIoU is lost when explicit correspondence between queries and spatial image regions is removed.This result supports spatially aligned region-query interactions in the network.
  • Ablation Study: 2% gIoU is gained by language cross-attention, while further region dependency modeling improves GRES feature fusion.The ablation attributes the first gain to region-word interaction and investigates region-region relationships with additional self-attention.
  • Results on GRES: Classic RES methods perform poorly on gRefCOCO, while adding ReLA to LAVT improves val-set cIoU by more than 4%.The comparison uses a common backbone and adapts two-stage methods for target-instance prediction and merging.
  • Results on GRES: Around 40% of no-target samples are missed despite ReLA’s higher N-acc., indicating that dedicated no-target identification remains challenging.T-acc. stays mostly above 95%, while mask-based identification performs similarly to other methods; deceptive expressions may resemble real instances.
  • Qualitative Results and Failure Cases: ReLA handles multiple targets, counting, shared attributes, and excluding relationships, but fails on deceptive possession and fine-grained contextual cases.Examples include confusing a dominant central laptop with the left girl’s laptop and accepting a nearby person as sitting on the bed.

6. Conclusion

The paper defines GRES and introduces gRefCOCO to address classic RES limitations for multi-target and no-target expressions. ReLA explicitly models region-region and region-language relationships and achieves state-of-the-art results on both GRES and classic RES.

  • Conclusion: GRES allows expressions to refer to an arbitrary number of targets, including multi-target and no-target cases excluded by classic RES.The benchmark is intended to reduce restrictions on language inputs and broaden RES application scope.
  • Conclusion: gRefCOCO is a large-scale dataset supporting expressions with arbitrary numbers of target objects.It provides the dataset foundation for research on the generalized task.
  • Conclusion: ReLA models relationships between image regions and words and consistently achieves state-of-the-art results on GRES and classic RES.The method is presented as a baseline designed for the generalized task.
  • Conclusion: GRES supports applications involving multiple instances, absent objects, and possible image-retrieval scenarios.The conclusion identifies these as expanded usage cases enabled by the generalized task definition.
Loading 2306.00968v1…