PaperlayerView exact revision on arXiv

Source-linked AI summary

Generalized Decoding for Pixel, Image, and Language

Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, Nanyun Peng, Lijuan Wang, Yong Jae Lee, Jianfeng Gao

arXiv:2212.11270v1cs.CVcs.CL

TL;DR

Existing vision systems commonly specialize by task or granularity, while pixel-level understanding also faces scarce annotations and limited open-vocabulary recognition. X-Decoder introduces a generalized decoder with latent and text queries, shared pixel- and token-level outputs, and joint training across segmentation and image-text data. It achieves strong zero-shot and finetuned transfer across segmentation and vision-language tasks, while supporting task interaction, efficient finetuning, and novel task compositions.

  • Problem

    Specialized models limit synergy across granularities, while pixel-level understanding has scarce annotations and remains less studied for open-vocabulary recognition.

  • Method

    X-Decoder uses latent and text queries with pixel-level masks and token-level semantics, sharing one text encoder and training end-to-end on segmentation and image-text data.

  • Results

    X-Decoder achieves strong zero-shot and task-specific transfer across segmentation and vision-language tasks, including competitive or state-of-the-art performance across reported settings.

  • Takeaways & Limitations

    The unified design supports cross-task interaction, efficient finetuning, and novel task compositions such as referring captioning and image editing.

Abstract

from arXiv · show

We present X-Decoder, a generalized decoding model that can predict pixel-level segmentation and language tokens seamlessly. X-Decodert takes as input two types of queries: (i) generic non-semantic queries and (ii) semantic queries induced from text inputs, to decode different pixel-level and token-level outputs in the same semantic space. With such a novel design, X-Decoder is the first work that provides a unified way to support all types of image segmentation and a variety of vision-language (VL) tasks. Further, our design enables seamless interactions across tasks at different granularities and brings mutual benefits by learning a common and rich pixel-level visual-semantic understanding space, without any pseudo-labeling. After pretraining on a mixed set of a limited amount of segmentation data and millions of image-text pairs, X-Decoder exhibits strong transferability to a wide range of downstream tasks in both zero-shot and finetuning settings. Notably, it achieves (1) state-of-the-art results on open-vocabulary segmentation and referring segmentation on eight datasets; (2) better or competitive finetuned performance to other generalist and specialist models on segmentation and VL tasks; and (3) flexibility for efficient finetuning and novel task composition (e.g., referring captioning and image editing). Code, demo, video, and visualization are available at https://x-decoder-vl.github.io.

1. Introduction

X-Decoder addresses the challenge of unifying pixel-level segmentation with image- and language-level vision tasks. Its generalized decoder and end-to-end training combine diverse queries, outputs, and supervision to support broad transfer across segmentation and vision-language tasks.

  • Motivation: Visual understanding spans image-, region-, and pixel-level tasks, but specialized model designs have limited synergy across these granularities.The paper identifies cross-granularity learning as a central motivation for general-purpose vision models.
  • Motivation: Pixel-level understanding remains difficult because segmentation annotations are scarce, open-vocabulary grouping is less studied, and cross-granularity learning is non-trivial.These constraints motivate combining segmentation data with image-text supervision.
  • Approach: X-Decoder uses generic non-semantic queries and language-aware queries to decode pixel-level masks and token-level semantics for different vision and vision-language tasks.The decoder combines query types and output types within a shared design.
  • Approach: A single text encoder represents textual inputs across segmentation concepts, referring phrases, captions, and questions, supporting a shared visual-semantic space.This shared encoding is intended to facilitate synergy across tasks.
  • Training: End-to-end pretraining unites panoptic segmentation, referring segmentation, and image-text pairs without pseudo-labeling from image-text data.X-Decoder directly proposes segmentation candidates and maps regions to caption content during training.
  • Results: X-Decoder transfers across segmentation and vision-language tasks in zero-shot and task-specific settings, while also supporting efficient finetuning and novel task compositions.The introduction reports state-of-the-art results on ten settings of seven datasets and consistent superiority after task-specific transfer.

2. From Specialist to Generalist Models

The paper positions X-Decoder within the shift from specialist to generalist models for pixel-level and vision-language understanding. It combines generic and referring segmentation with vision-language tasks in one generalized decoder and emphasizes task interaction as a distinguishing capability.

  • Pixel-Level Understanding: Pixel-level understanding includes semantic, instance, and panoptic segmentation, which differ in whether pixels receive semantic labels, object-instance grouping, or both.These tasks represent distinct forms of pixel-level and instance-level understanding.
  • Pixel-Level Understanding: Unified segmentation models such as Mask2Former still generally recognize only a limited number of categories, restricting open-vocabulary recognition.The related-work discussion contrasts closed-set unified segmentation with open-vocabulary approaches.
  • Open-Vocabulary Segmentation: Open-vocabulary segmentation methods transfer or distill visual-semantic knowledge from foundation models, while other approaches learn language-image grouping directly.The paper cites CLIP- and ALIGN-based transfer as well as language-image pretraining from scratch.
  • Referring Segmentation: Referring segmentation is inherently open-vocabulary because it does not assume a fixed set of phrases during training or inference.Existing models typically specialize in referring segmentation and use multimodal fusion strategies.
  • Unified Modeling: X-Decoder is presented as the first model to combine generic and referring segmentation while jointly learning from segmentation data and image-text pairs end-to-end.The stated goal is richer pixel-level and image-level understanding through cross-task synergy.
  • Vision-Language Understanding: Vision-language pretraining uses image-text data to support tasks such as retrieval, captioning, and VQA, with prior work progressing from fusion models toward end-to-end transformers.The paper situates X-Decoder among approaches that combine image-text pairs with region-level data.
  • From Specialist to Generalist: Generalist-model research has explored multitask learning and sequential decoding, but task interactions and performance trade-offs remain important concerns.The paper motivates a unified design that explicitly addresses interaction across tasks.

3. X-Decoder

X-Decoder unifies segmentation, retrieval, captioning, and VQA through shared query-driven decoding of pixel-level masks and token-level semantics. A common text encoder and task-specific query combinations support joint pretraining and adaptation without extra heads.

  • 3.1. Formulation: X-Decoder encodes images and text, then decodes latent and text queries into pixel-level masks and token-level semantic outputs.Latent queries support generic vision tasks, while text queries make the decoder language-aware; their combinations support language-conditioned tasks.
  • 3.2. Unification of Tasks: The same decoder supports generic segmentation, referring segmentation, image-text retrieval, image captioning, and VQA through different query and output combinations.Generic segmentation uses latent queries; referring segmentation uses latent and text queries; retrieval uses latent queries; captioning and VQA use both query types without mask prediction.
  • 3.1. Formulation: A single text encoder represents segmentation concepts, referring phrases, captions, and VQA questions across tasks.Class names are converted into prompted phrases for segmentation, while the shared encoder processes textual inputs from the other tasks.
  • 3.4. End-to-End Pre-training: X-Decoder jointly pretrains on panoptic segmentation, referring segmentation, and image-text pairs, then finetunes individual tasks without extra heads.The method directly proposes meaningful segmentation candidates from image-text pairs instead of using pseudo-labeling to extract fine-grained supervision.
  • 3.3. Unified Architecture: Its transformer decoder cross-attends queries to hierarchical visual features and self-attends among latent and text queries to promote task synergy.The self-attention design uses latent queries for global image representation and segmentation, text-query context for captioning, and text information for referring segmentation.
  • 3.4. End-to-End Pre-training: Semantic training uses task-specific losses, including language-image contrastive learning for retrieval and text-concept matching for open-vocabulary mask classification.Mask classification compares decoder outputs from latent queries with embeddings of class names, including background.

4. Experiments

Experiments evaluate X-Decoder under task-specific transfer, zero-shot open-vocabulary segmentation, ablations, and task composition. The results show competitive or superior performance across segmentation and vision-language tasks while preserving cross-task interaction.

  • Experimental Setup: 104k segmentation images and 4M image-text pairs are used for pretraining, alongside referring segmentation data from COCO2017.The segmentation pretraining set contains 30k images with referring segmentation annotations.
  • Task-Specific Transfer: 51.0 PQ on ADE Panoptic Segmentation establishes a new state of the art for finetuned X-Decoder (L).On COCO, performance is comparable to Mask2Former and kMaX-DeepLab; the tiny model reaches 61.9 versus LAVT-Base at 61.2 on COCO G-ref.
  • Task-Specific Transfer: X-Decoder achieves competitive vision-language transfer, outperforming UNITER on COCO retrieval and VinVL by 1.3 CIDEr and 1.7 BLEU.It also beats all compared methods on Flickr30k retrieval while using fully decoupled image and text encoders.
  • Zero-Shot Transfer: New state-of-the-art results are reported on 10 settings of 7 datasets across panoptic, instance, and semantic segmentation.The model applies directly without changing model weights, using PQ, mAP, and mIoU for the three segmentation types.
  • Ablation Studies: Removing referring segmentation lowers COCO g-Ref by 2.0 points and CIDEr by 3.2 points, indicating benefits from joint text-query training.The shared text encoder supports interaction between referring segmentation and captioning.
  • Task Composition: The unified model supports region-based retrieval, referring captioning, and referring image editing without architecture or weight changes.Task composition combines retrieval, localization, captioning, and diffusion-based editing through the shared model.

5. Conclusion

The paper presents X-Decoder as a generalized model for pixel-level and image-level vision-language understanding. Its unified design supports segmentation and VL tasks with strong generalizability and competitive or state-of-the-art performance.

  • 5. Conclusion: X-Decoder unifies generic segmentation, referring segmentation, and vision-language tasks in one generalized design.The model is intended to support pixel-level and image-level visual understanding.
  • 5. Conclusion: The reported results demonstrate strong generalizability and competitive or state-of-the-art performance across the supported tasks.

A. Experiment Settings

The experiments use distinct pretraining and finetuning configurations for retrieval, captioning, VQA, generic segmentation, and referring segmentation. These settings vary image resolution, batch size, learning rates, and training schedules by task.

  • Image-Text Retrieval: Image-text retrieval is finetuned for 10 epochs at resolution 384 with batch size 2048.Learning rates are 3e-5 for X-Decoder and 3e-6 for the vision and language backbones.
  • Image Captioning: Image captioning is finetuned for 10 epochs at resolution 480 with batch size 256 and beam size 5.CIDEr optimization is not used during captioning finetuning.
  • VQA: VQA adds a classification layer and finetunes for 10 epochs at resolution 640 with batch size 256.The VQA classification layer uses learning rate 1e-3.
  • Segmentation: Generic and referring segmentation are each finetuned for 24 epochs with learning-rate decay at epochs 21 and 23.Generic segmentation uses batch sizes 64 on ADE20K and 32 on COCO; referring segmentation uses batch size 64.

B. Open-Vocab Segmentation Benchmark

The paper introduces a nine-dataset open-vocabulary segmentation benchmark spanning vocabulary sizes, domains, and segmentation tasks, with standardized evaluation metrics.

  • The benchmark covers 9 datasets with different evaluation metrics for open-vocabulary segmentation.
  • It supports semantic, instance, and panoptic segmentation across scopes ranging from 20 to 847 classes.
  • Its evaluation scenes include common objects, indoor scenes, and autonomous-driving scenarios.
  • Table 9 compares finetuning results between X-Decoder and X-Decoder-VL pretrained only on 4M image-text pairs.

C.1. Complementariness between Vision and VL

The ablations examine how segmentation and vision-language data contribute to X-Decoder and how architectural choices affect zero-shot performance.

  • Complementariness between Vision and VL: Removing segmentation data tests whether vision-language pretraining objectives benefit image segmentation and whether segmentation objectives benefit vision-language understanding.
  • Model Architecture Inspection: Table 10 compares Swin, FocalNet, and DaViT backbones in zero-shot settings without task-specific finetuning.
  • Open-Vocabulary Generic Segmentation Settings Inspection: Table 11 reports additional open-vocabulary segmentation results for models pretrained with COCO segmentation and caption annotations and models using different large vision backbones.

C.2. Model Architecture Inspection

The architecture inspection studies backbone choice and progressively enriched training settings for open-vocabulary segmentation.

  • Model architecture: The study compares Swin, FocalNet, and DaViT architectures, including multi-scale deformable attention used in Mask2Former.
  • Training settings: X-Decoder-Seg adds a learnable text encoder to Mask2Former, while X-Decoder-Seg+ uses caption nouns for Hungarian matching to enlarge vocabulary.
  • Training settings: Adding extra image-text pairs improves open-vocabulary segmentation, especially for large vocabulary settings such as ADE-150 and CONTEXT-59/459.
  • Backbone pretraining: Backbone pretraining also matters: similar-sized backbones differ when pretrained on ImageNet-21K versus hundreds of millions of image-text pairs.

D. Segmentation In the Wild Benchmark

The SegInW evaluation tests X-Decoder across diverse visual domains and tuning regimes, while qualitative studies extend its zero-shot use to video, regional captioning, and image editing.

  • SegInW benchmark: SegInW evaluates zero-shot, few-shot, and fine-tuned segmentation for five models across three tuning scales.
  • Zero-shot performance: X-Decoder achieves higher mAP on common objects but lower mAP on fine-grained toolkits and rare concepts such as rail and brain tumor.
  • Tuning strategies: X-Decoder performs much better than X-Decoder-Seg+ in few-shot and finetuning settings, although the gap shrinks when many parameters are tuned.
  • Tuning strategies: Tuning class embeddings produces similar or better results than tuning the full decoder for 0-10 shots on larger backbones.
  • Video generalization: Zero-shot generic segmentation generalizes to YoutubeVOS with predictions consistent across video frames.
  • Video generalization: Without architectural change or finetuning, X-Decoder generates referring video segmentation outputs from referring phrases.
  • Task composition: The model simultaneously segments referred regions and generates regional captions, despite never being trained specifically for regional caption generation.

F. Discussions

The discussion identifies future directions for more unified and efficient pretraining, while warning that web-crawled image-text data may contain offensive or biased content. The supplied figure and table captions indicate broad task coverage and parameter-efficient tuning settings, but do not report their outcomes.

  • Future Directions: Future work could unify CLIP-style image-text pretraining with decoder pretraining in one end-to-end stage.The current model separately pretrains the image and text encoders.
  • Future Directions: Future work could jointly learn from pixel-level, region-level, and image-level supervision despite differing annotation costs.Pixel-level segmentation annotations are described as substantially scarcer because they are costly.
  • Social Impact: Large-scale web-crawled image-text pairs may introduce offensive or biased content, requiring checks of training data and model predictions before deployment.The paper frames this as a potential negative impact of the data used for pretraining.
  • Efficient Finetuning: The appendix captions describe SegInW tuning settings that vary the tuned components and parameter counts, including 39.3M, 1.15M, and 0.26M parameters.The supplied captions do not state the corresponding performance values.
  • Qualitative Results: The reported materials include zero-shot examples spanning generic and referring video segmentation, image captioning, referring captioning, and referring image inpainting.These are identified as figures sourced from YoutubeVOS videos, COCO 2017 validation images, and web images.
Loading 2212.11270v1…