PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Modal Progressive Comprehension for Referring Segmentation

Si Liu, Tianrui Hui, Shaofei Huang, Yunchao Wei, Bo Li, Guanbin Li

arXiv:2105.07175v1cs.CVcs.MM

TL;DR

Referring segmentation requires pixel-level masks for entities specified by natural-language expressions, while prior approaches mainly use one-stage multimodal fusion. The paper proposes progressive comprehension through CMPC-I and CMPC-V, adds text-guided feature exchange, and reports state-of-the-art performance on seven benchmarks. A stated limitation is that models cannot generate particularly fine masks on unseen samples.

  • Problem

    Prior referring segmentation methods use one-stage implicit visual-linguistic fusion, whereas the task requires distinguishing the referred entity among candidates.

  • Method

    CMPC progressively uses entity, attribute, relational, and action words with spatial or temporal graph reasoning, while TGFE exchanges multi-level multimodal features under textual guidance.

  • Results

    The combined frameworks achieve state-of-the-art performance on four referring image segmentation benchmarks and three referring video segmentation benchmarks.

  • Takeaways & Limitations

    Progressive comprehension provides a common framework for referring segmentation across image and video modalities.

  • Takeaways & Limitations

    On unseen HMDB Sentences samples, models cannot generate particularly fine masks.

Abstract

from arXiv · show

Given a natural language expression and an image/video, the goal of referring segmentation is to produce the pixel-level masks of the entities described by the subject of the expression. Previous approaches tackle this problem by implicit feature interaction and fusion between visual and linguistic modalities in a one-stage manner. However, human tends to solve the referring problem in a progressive manner based on informative words in the expression, i.e., first roughly locating candidate entities and then distinguishing the target one. In this paper, we propose a Cross-Modal Progressive Comprehension (CMPC) scheme to effectively mimic human behaviors and implement it as a CMPC-I (Image) module and a CMPC-V (Video) module to improve referring image and video segmentation models. For image data, our CMPC-I module first employs entity and attribute words to perceive all the related entities that might be considered by the expression. Then, the relational words are adopted to highlight the target entity as well as suppress other irrelevant ones by spatial graph reasoning. For video data, our CMPC-V module further exploits action words based on CMPC-I to highlight the correct entity matched with the action cues by temporal graph reasoning. In addition to the CMPC, we also introduce a simple yet effective Text-Guided Feature Exchange (TGFE) module to integrate the reasoned multimodal features corresponding to different levels in the visual backbone under the guidance of textual information. In this way, multi-level features can communicate with each other and be mutually refined based on the textual context. Combining CMPC-I or CMPC-V with TGFE can form our image or video version referring segmentation frameworks and our frameworks achieve new state-of-the-art performances on four referring image segmentation benchmarks and three referring video segmentation benchmarks respectively.

1 INTRODUCTION

The paper frames referring segmentation as pixel-level localization of entities specified by open-ended expressions and proposes progressive multimodal reasoning instead of one-stage fusion. CMPC-I and CMPC-V progressively use entity, attribute, relational, and action words, while TGFE exchanges multi-level features; the resulting frameworks report state-of-the-art performance across seven benchmarks.

  • Referring segmentation produces pixel-level masks for entities described by natural-language expressions in images or videos.
  • Previous methods mainly fuse visual and linguistic features through one-stage implicit mechanisms, whereas human comprehension first locates candidate entities and then reasons over relationships.
  • CMPC-V extends the image process with action-aware temporal graph reasoning to use video context when static relationships are insufficient.
  • TGFE uses textual guidance to exchange and refine multimodal features across backbone levels before mask prediction.
  • CMPC progressively aligns multimodal features using informative words, first perceiving candidate entities and then reasoning to identify the referent.
  • CMPC-I uses entity and attribute words for entity perception, followed by spatial graph reasoning with relational words to highlight the referent and suppress mismatched entities.
  • The combined image and video frameworks achieve state-of-the-art performance on four referring image segmentation benchmarks and three referring video segmentation benchmarks.

2 RELATED WORK

Related work covers semantic segmentation, referring expression grounding and segmentation, multimodal feature fusion, attention-based context extraction, and graph-based reasoning. The paper positions its approach within this progression toward multimodal and relational context modeling.

  • Semantic segmentation architectures such as FCN, DeepLab, and PSPNet use convolutional designs and multi-scale visual context for pixel classification.
  • Referring expression grounding localizes expression-matched entities, commonly at bounding-box level, using cross-modal correlation and graph-based models.
  • Referring image segmentation extends localization to precise masks by fusing visual and linguistic features with CNN-LSTM, multimodal LSTM, recurrent refinement, and attention mechanisms.
  • Graph-based approaches model contextual interactions among visual, temporal, query, and answer nodes through message passing.

3 METHOD

The method section introduces CMPC-I and CMPC-V as image and video instantiations of the proposed Cross-Modal Progressive Comprehension scheme.

  • The paper instantiates CMPC for referring segmentation on image and video data.
  • The image implementation is denoted CMPC-I.
  • The video implementation is denoted CMPC-V.

3.1 CMPC on Image

CMPC-I progressively narrows image candidates by first perceiving entities and attributes, then using relational cues for spatial graph reasoning. The resulting multimodal features are further text-guided and fused across levels for mask prediction.

  • Image and Words Feature Extraction: Image features and word features are extracted at multiple backbone levels and transformed for multimodal processing.The image features correspond to CNN stages 3, 4, and 5, while word features are produced by a language encoder.
  • Entity Perception: Entity and attribute features are bilinearly fused with spatial image features to perceive all candidate entities matching the expression.The resulting multimodal feature integrates visual and linguistic context for each spatial region.
  • Entity Perception: Entity Perception classifies words into entity, attribute, relation, and unnecessary categories without direct word-type annotations.Word classification is supervised through the final binary cross-entropy segmentation loss.
  • Relation-Aware Reasoning: Relation-Aware Reasoning builds a fully connected spatial graph whose vertices represent image regions and whose edge weights depend on affinities with relational words.Relational words act as routers for adaptive communication between spatial regions.
  • Relation-Aware Reasoning: Graph convolution selectively highlights the referent using relationship cues while suppressing irrelevant regions, producing more discriminative multimodal features.The enhanced features are reshaped and combined with aggregated textual context to form the output feature for prediction.

3.2 CMPC on Video

CMPC-V extends image-based progressive comprehension to video by using action cues and temporal reasoning. It aggregates action-matched features across frames, reasons over a temporal graph, and projects the result to the center frame.

  • Video Feature Extraction: The video model processes a K-frame clip with a CNN backbone and extracts multi-level features using the same word-feature protocol as the image model.The center frame carries the annotated pixel-wise mask, and coordinate features are fused at each visual level.
  • Action-Aware Reasoning: Video expressions are classified into entity, attribute, relation, action, and unnecessary word categories to construct a global action feature.The action feature is a weighted combination of all expression-word features.
  • Action-Aware Reasoning: Entity Perception runs independently on every frame, while Relation-Aware Reasoning is performed only on the annotated center frame to reduce computation.The resulting video multimodal features preserve frame-wise entity candidates and center-frame static relationship reasoning.
  • Action-Aware Reasoning: Dot-product attention between video features and the action feature produces dense attention maps that aggregate global action-matched features for every frame.These frame-level features become vertices of a temporal graph.
  • Action-Aware Reasoning: Graph convolution reasons over the fully connected temporal graph and projects the reasoned context back onto center-frame spatial features.The projected temporal context is fused with center-frame multimodal features and repeated sentence context for the referent feature.

3.3 Text-Guided Feature Exchange

TGFE enables multi-level visual features to exchange information under textual guidance. Repeated exchanges refine each level before ConvLSTM produces the final mask prediction.

  • Module Overview: TGFE takes multimodal features from levels 3, 4, and 5 together with a sentence feature to communicate information across visual levels.The module performs n rounds of feature exchange.
  • Text-Guided Exchange: At each exchange round, every level attends to the sentence and its spatial features to derive a text-conditioned context vector.Sentence and globally pooled visual context are fused before selecting information from other levels.
  • Text-Guided Exchange: Each level selects relevant features from the other two levels through channel attention and adds them to its original features.The exchange is repeated across levels and rounds under textual guidance.
  • Output Fusion: After n exchange rounds, the refined multi-level features are fused with ConvLSTM to produce the final mask prediction.

4.1 Experimental Setup

The evaluation covers four referring image segmentation benchmarks and three referring video segmentation benchmarks. Performance is measured with overlap- and precision-based metrics, using fixed implementation settings and mask refinement.

  • Benchmarks: The study evaluates image segmentation on UNC, UNC+, G-Ref, and ReferIt, and video segmentation on A2D Sentences, J-HMDB Sentences, and Refer-Youtube-VOS.
  • Datasets: UNC, UNC+, and G-Ref are collected on MS-COCO, while ReferIt is collected on IAPR TC-12 with distinct expression and object characteristics.UNC+ contains no location words, and G-Ref expressions are longer than those in the other listed datasets.
  • Experimental Comparison: Table 1 compares state-of-the-art methods across four image datasets using overall IoU, with 3 TGFE exchange rounds.The table marks MAttNet as n/a when it uses a different split.
  • Evaluation Metrics: Image models are evaluated with Prec@X and overall IoU, while video models additionally use mAP and mean IoU.Prec@X measures the percentage of samples whose IoU exceeds thresholds from 0.5 through 0.9.
  • Implementation: DeepLab-ResNet101 provides fixed CNN-backbone features, inputs are resized to 320 × 320, and video clips contain 5 frames for CMPC-V.Training uses pixel-averaged binary cross-entropy, and DenseCRF refines masks for comparison with prior work.

4.2 Comparison with State-of-the-arts

The proposed models outperform prior methods across referring image and video segmentation benchmarks, including cross-dataset evaluation and settings using only RGB input. Results also indicate stronger generalization and competitive performance without extensive temporal memory interactions.

  • Referring Image Segmentation: 1.45%-3.49% performance gains over STEP are achieved consistently across four referring image segmentation datasets.The largest gain is 3.49% IoU on the G-Ref validation set.
  • Referring Video Segmentation: The A2D Sentences comparison reports significant improvements over state-of-the-art methods using only RGB input.The table compares referring video segmentation methods on A2D Sentences.
  • Referring Video Segmentation: 3D visual features improve the proposed video model over its 2D counterpart, while the method surpasses PRPE on most A2D Sentences metrics.Overall IoU remains comparable with PRPE.
  • Referring Video Segmentation: 4.0% Overall IoU and 5.7% Prec@0.5 gains over previous methods are reported on JHMDB Sentences without finetuning.All methods obtain only 0.0% or 0.1% on Prec@0.9 for unseen samples.
  • Referring Video Segmentation: The video model outperforms URVOS on most Refer-Youtube-VOS metrics without memory attention between frames, except Prec@0.5.This evaluation uses a model trained on Refer-Youtube-VOS for 200,000 iterations.

4.3 Ablation Studies

Ablations show that progressive comprehension stages, TGFE, and multimodal feature exchange each contribute to performance, while graph depth requires dataset-dependent tuning. Temporal action-aware reasoning is especially valuable for video segmentation.

  • Components of CMPC-I Module: 8.02% IoU improvement over baseline is obtained by combining entity perception and relation-aware reasoning in CMPC-I.Entity perception alone improves IoU by 1.70%, while relation-aware reasoning alone improves it by 6.04%.
  • Components of CMPC-I Module: 9.76% performance gain is reached when CMF is added, while TGFE substantially improves performance and ConvLSTM provides only marginal gains.CMF concatenates multimodal features from entity perception instead of pure visual features.
  • TGFE Module: TGFE improves CMPC-I consistently with multi-level visual features and enables feature exchange across backbone levels.The multi-level ablation uses a single round of feature exchange in TGFE.
  • Components of CMPC-V Module: The AAR stage produces a large gain beyond TGFE, EP, and RAR, demonstrating the importance of temporal context for referring video segmentation.CMPC-V extends CMPC-I with action-aware reasoning based on temporal information.
  • Components of CMPC-V Module: Direct relevance outperforms adaptive relevance for constructing AAR adjacency matrices, making direct propagation more suitable across frames.AR derives adjacency similarly to RAR, whereas DR uses direct feature relevance.
  • Number of Graph Convolution Layer: More graph convolution layers can degrade performance, although two layers help on G-Ref relative to one layer.The authors associate this dataset dependence with G-Ref's longer referring expressions and caution that excessive layers introduce noise.

4.4 Visualization Analyses

Visualization analyses show that the model’s word classification, progressive affinity responses, relation reasoning, and action reasoning contribute to identifying the referred entities in images and videos.

  • Accuracy of Word Classification: Random word-type assignments cause notable Overall IoU degradation across four referring image segmentation datasets.Table 11 uses Overall IoU to compare the original model with randomly assigned word classification probabilities.
  • Accuracy of Word Classification: Random word categories can make the model select an incorrect object, whereas the original categories identify the referred object.In the illustrated example, modifying the categories causes the model to misrecognize the middle man.
  • Qualitative Results: The full image and video models produce clearer referent masks than their baseline models in qualitative comparisons.Examples show improved distinction among girls, coherent baby and dog masks, and better handling of attributes and location relationships.
  • Visualization of Affinity Maps: Affinity maps become more concentrated on the referent as expressions progress from entity words to full sentences.When expressions are manually changed, the high-response area shifts to the newly described entity.
  • Relation- and Action-Aware Reasoning: Relation-aware reasoning helps distinguish the correct referent from other visually similar entities in image examples.Compared with entity perception alone, relation-aware reasoning correctly selects the rightmost boy and other relationally specified targets.
  • Relation- and Action-Aware Reasoning: Action-aware reasoning enables the video model to locate the person matching an action cue when relation-aware reasoning alone fails.The full model identifies the man moving his head up and down, while the model without action-aware reasoning selects incorrectly.

5 CONCLUSION AND FUTURE WORK

The paper concludes that CMPC progressively uses entity, attribute, relational, and action words with graph reasoning, while TGFE exchanges text-guided multi-level visual features. The resulting image and video frameworks consistently outperform previous state-of-the-art methods across seven benchmarks, and future work targets more structured linguistic analysis and compact graphs.

  • Conclusion: CMPC first perceives candidate entities, then uses relational and action words with graph reasoning to highlight the referent and suppress others.CMPC-I handles image inputs, while CMPC-V extends the process to video inputs.
  • Conclusion: TGFE selectively integrates multi-level visual features under textual guidance to refine mask prediction.The module allows features from different visual-backbone levels to communicate and be mutually refined.
  • Conclusion: The proposed frameworks consistently outperform previous state-of-the-art methods on four referring image segmentation and three referring video segmentation benchmarks.The conclusion reports performance across four image benchmarks and three video benchmarks.
  • Future Work: Future work will analyze linguistic information more structurally and explore more compact graph formulations.The paper also states that its code is publicly available.
Loading 2105.07175v1…