Source-linked AI summary
PolyFormer: Referring Image Segmentation as Sequential Polygon Generation
Jiang Liu, Hui Ding, Zhaowei Cai, Yuting Zhang, Ravi Kumar Satzoda, Vijay Mahadevan, R. Manmatha
TL;DR
Referring image segmentation must localize objects described by language, but dense masks and independently predicted pixels do not directly capture structured contours. PolyFormer recasts localization as autoregressive sequence generation with a regression decoder for continuous coordinates, achieving strong RIS results and generalizing to referring video segmentation without finetuning.
Problem
Referring image segmentation requires localizing a language-described object, while dense pixel prediction neglects relationships among output pixels and structured polygon prediction is difficult.
Method
PolyFormer uses a multimodal sequence-to-sequence framework to autoregressively generate polygon vertices and bounding-box corners, with a regression-based decoder producing continuous coordinates without quantization.
Results
76.94%, 72.15%, and 71.15% mIoU on RefCOCO, RefCOCO+, and RefCOCOg improve over the state of the art by 2.48%, 5.40%, and 4.52%, respectively; Ref-DAVIS17 reaches 61.5% J &F without finetuning.
Takeaways & Limitations
PolyFormer shows that polygon-based sequential prediction can outperform mask-based methods across three RIS benchmarks and generalize to unseen video and synthetic scenarios.
Abstract
from arXiv · showhide
In this work, instead of directly predicting the pixel-level segmentation masks, the problem of referring image segmentation is formulated as sequential polygon generation, and the predicted polygons can be later converted into segmentation masks. This is enabled by a new sequence-to-sequence framework, Polygon Transformer (PolyFormer), which takes a sequence of image patches and text query tokens as input, and outputs a sequence of polygon vertices autoregressively. For more accurate geometric localization, we propose a regression-based decoder, which predicts the precise floating-point coordinates directly, without any coordinate quantization error. In the experiments, PolyFormer outperforms the prior art by a clear margin, e.g., 5.40% and 4.52% absolute improvements on the challenging RefCOCO+ and RefCOCOg datasets. It also shows strong generalization ability when evaluated on the referring video segmentation task without fine-tuning, e.g., achieving competitive 61.5% J&F on the Ref-DAVIS17 dataset.
1. Introduction
PolyFormer addresses the challenge of structured geometric localization in referring image segmentation by generating polygon vertices sequentially rather than predicting dense masks. Its regression-based decoder outputs continuous coordinates, and experiments report gains across major benchmarks plus transfer to video segmentation.
- Referring image segmentation localizes an object’s segmentation mask from a natural-language query, extending segmentation beyond predefined categories.
- Structured polygon representations are cheaper than dense masks but remain difficult to predict effectively, especially with convolutional networks.
- PolyFormer formulates referring image segmentation and referring expression comprehension as sequence-to-sequence prediction, generating polygon vertices and bounding-box corner points autoregressively.It fuses image patches and text query tokens as the input sequence, while each vertex conditions on previously generated vertices.
- 76.94%, 72.15%, and 71.15% mIoU on RefCOCO, RefCOCO+, and RefCOCOg correspond to absolute gains of 2.48%, 5.40%, and 4.52% over the state of the art.PolyFormer also achieves 61.5% J &F on Ref-DAVIS17 without finetuning, comparable with a task-specific method.
- The regression-based decoder predicts continuous 2D coordinates directly instead of quantized classifications, avoiding coordinate quantization error.This design targets the continuous nature of geometric localization.
2. Related Work
Prior work studies multimodal feature extraction and fusion for RIS and REC, while PolyFormer places both tasks in a unified sequence-to-sequence framework. This connects contour-based structured prediction with broader seq2seq modeling trends in vision-language research.
- RIS research has explored CNN, recurrent, and transformer feature extractors alongside concatenation and attention-based multimodal fusion.
- REC methods include two-stage region-proposal ranking and one-stage direct box prediction, with several works jointly learning REC and RIS.
- PolyFormer differs from prior multitask point-prediction work by reporting improved performance when jointly learning RIS and REC.
- Seq2seq modeling, established in NLP for tasks such as translation and question answering, has also motivated unified task formulations in vision and vision-language.
- Contour-based instance segmentation represents object masks through polygons or deformable contours instead of dense pixel masks.
3. PolyFormer
PolyFormer formulates referring image segmentation as autoregressive polygon generation rather than dense mask prediction. It combines multimodal encoding with a regression-based decoder that predicts continuous coordinates and uses polygon augmentation during training.
- 3. PolyFormer: PolyFormer encodes images and referring expressions, then sequentially generates bounding-box corners and polygon vertices instead of dense segmentation masks.Visual and textual features are projected into a shared space, concatenated, and processed by a multimodal transformer encoder.
- 3.2. Target Sequence Construction: Polygon vertices are represented in clockwise order, beginning with the vertex closest to the image origin; <SEP>, <BOS>, and <EOS> organize polygon sequences.Multiple polygons are ordered by the distance between their starting points and the image origin.
- 3.4. Regression-based Transformer Decoder: The regression-based decoder predicts continuous floating-point coordinates directly, avoiding the quantization error introduced by discretized coordinate classification.A 2D coordinate codebook with bilinear interpolation provides coordinate embeddings for arbitrary floating-point positions.
- 3.4. Regression-based Transformer Decoder: A class head predicts whether each output is a coordinate, separator, or end-of-sequence token, while a coordinate head predicts bounding-box and polygon coordinates.The decoder uses transformer layers with self-attention, cross-attention, and feed-forward processing before the two prediction heads.
- 3.5. Training: Polygon augmentation samples sparse polygons at varied granularities from interpolated dense contours, increasing polygon diversity and reducing overfitting to one representation.During inference, generation begins with <BOS>, stops at <EOS>, and converts the generated polygon predictions into the final segmentation mask.
4. Experimental Results
PolyFormer is evaluated across referring image segmentation benchmarks, referring expression comprehension, and zero-shot video segmentation, with ablations examining its decoder and target-sequence design. It achieves strong benchmark performance and transfers to video without fine-tuning.
- Referring image segmentation: PolyFormer models outperform previous methods on each split of the three RIS datasets under all reported metrics.PolyFormer-B gains 1.5%, 0.2%, and 2.28% mIoU over LAVT across RefCOCO splits, while larger PolyFormer-L adds roughly 1–2 points over PolyFormer-B.
- Referring image segmentation: PolyFormer-B exceeds RefTr by 3.9%, 3.93%, and 5.24% absolute mIoU on RefCOCO+ validation, test A, and test B.On RefCOCOg, it improves over RefTr by 2.73% on validation and 2.49% on the test set.
- Zero-shot video transfer: 61.5% J &F is achieved on Ref-DAVIS17 without video fine-tuning, exceeding the fully video-trained ReferFormer.The model applies frame-by-frame by treating video as a sequence of images.
- Ablation studies: Regression consistently outperforms coordinate classification, including +1.86 for RIS and +2.38 for REC on RefCOCO+.The classification baseline quantizes coordinates into discrete bins, whereas the regression-based model predicts coordinates directly.
- Ablation studies: Polygon ordering improves RefCOCO mIoU by 12.43%, while polygon augmentation, multitask learning, and <SEP> add 3.72%, 3.07%, and 0.82%.Randomly ordered polygons achieve 55.92% mIoU, and similar trends appear on the other two datasets.
5. Conclusion
PolyFormer is a unified sequence-to-sequence framework for referring image segmentation and referring expression comprehension, with continuous-coordinate decoding and reported generalization to unseen scenarios. The authors identify accurate annotations as a remaining limitation and broader-impact considerations as requiring further study.
- Conclusion: PolyFormer unifies referring image segmentation and referring expression comprehension in a sequence-to-sequence framework that fuses multimodal inputs and supports multitask outputs.It is designed as a simple framework that may extend beyond these tasks.
- Conclusion: The regression-based decoder generates continuous 2D coordinates without quantization errors.
- Conclusion: PolyFormer achieves competitive results for RIS and REC and generalizes to unseen scenarios.
- Limitations and Broader Impacts: Training requires accurate bounding box and polygon annotations, motivating further exploration of weakly supervised data.
- Limitations and Broader Impacts: The broader impacts of the data and model require further study, including fairness, social bias, and potential misuse.
A. Additional Dataset Details
The experiments use four referring image datasets and Ref-DAVIS17 for referring video object segmentation, with distinct collection procedures, expression characteristics, and evaluation conventions. The supplied table caption identifies an ablation study on 2D coordinate-codebook size.
- Datasets: PolyFormer is evaluated on RefCOCO, RefCOCO+, RefCOCOg, ReferIt, and Ref-DAVIS17.The first four are image datasets; Ref-DAVIS17 supports referring video object segmentation.
- RefCOCO/RefCOCO+: RefCOCO and RefCOCO+ use a two-player-game collection process and contain roughly 142,000 referring expressions each.Their test partitions distinguish images with multiple people from images with multiple instances of other objects.
- RefCOCOg: RefCOCOg contains longer, more complex expressions averaging 8.4 words, compared with 3.5 words for RefCOCO and RefCOCO+.This makes RefCOCOg particularly challenging according to the passage.
- ReferIt: ReferIt includes more stuff segmentation masks, such as sky and ground, than the RefCOCO datasets.
- Ref-DAVIS17: Ref-DAVIS17 contains 90 videos, 1,544 referring expressions, and 205 objects, with Region Jaccard, Boundary F measure, and their average as evaluation metrics.Its videos are split into 60 for training and 30 for validation.
- Additional Experiments: The supplied table is an ablation study on the size of the 2D coordinate codebook.
B. Additional Implementation Details
The implementation projects image and language features into a shared 768-dimensional space and uses transformer attention with specified architectural settings. PolyFormer-B and PolyFormer-L differ in image-feature dimensionality.
- Feature Dimensions: PolyFormer-B uses 1024-dimensional image features, whereas PolyFormer-L uses 1536-dimensional image features.
- Feature Dimensions: Language features and coordinate embeddings have dimension 768, and image and language features are projected into the same 768-dimensional space.
- Transformer Settings: The transformer uses 12 self-attention and cross-attention heads, GELU activations, and label smoothing with factor 0.1 for L_cls.
C. Additional Experiment Results
An ablation varies the size of PolyFormer’s 2D coordinate codebook to study its effect on coordinate embeddings. The 64 × 64 configuration is selected as the default because it achieves the best result.
- Coordinate Codebook: The ablation changes the number of bins along the height and width dimensions of the 2D coordinate codebook.
- Coordinate Codebook: The coordinate codebook provides embeddings for discretized 2D spatial locations.The passage defines separate height and width bin counts and an embedding dimension.
- Coordinate Codebook: 64 × 64 bins achieve the best result and are adopted by default in the remaining experiments.
D.1. Cross-attention Map
During polygon generation, PolyFormer’s cross-attention concentrates on the referred object and moves around its boundary as vertices are predicted.
- Cross-attention concentrates on the object referred to by the sentence during polygon generation.
- The attention map moves around the object boundary as polygon vertices are generated.
- The visualization links each attention map to vertex prediction at time step t_poly.
D.2. Prediction Visualization
PolyFormer segments referred objects in challenging RefCOCOg and synthetic-image scenarios, while demonstrating generalization to unseen images and text descriptions. The comparisons include LAVT and SeqTR, with task-specific outputs generated for SeqTR.
- Prediction Visualization: PolyFormer segments referred objects with occlusion, complex shapes, partial visibility, or complex language descriptions.
- Prediction Visualization: PolyFormer demonstrates good generalization on synthetic images and text descriptions unseen during training.
- Prediction Visualization: On these unseen synthetic examples, LAVT and SeqTR fail to generate satisfactory results.
- Prediction Visualization: The visualization includes the referring expression “a without hairy brown color teddy bear.”
- Prediction Visualization: Figure 10 compares LAVT, SeqTR, and PolyFormer on the RefCOCOg test set.
- Prediction Visualization: PolyFormer simultaneously predicts the bounding box and polygon vertices that form the segmentation mask.