PaperlayerView exact revision on arXiv

Source-linked AI summary

PixelLM: Pixel Reasoning with Large Multimodal Model

Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, Xiaojie Jin

arXiv:2312.02228v3cs.CV

TL;DR

Existing multimodal models and reasoning segmentation methods have limited support for pixel-level masks involving multiple open-set targets, while prior approaches may require costly external segmentation models. PixelLM addresses this with a lightweight pixel decoder, segmentation codebook, target refinement loss, and the MUSE benchmark. It achieves state-of-the-art results across multiple segmentation benchmarks, with reported computational-cost reductions of up to 50%.

  • Problem

    Pixel-level reasoning segmentation remains limited for image tasks involving multiple open-set targets, and prior methods may depend on costly external segmentation models.

  • Method

    PixelLM uses a lightweight pixel decoder and segmentation codebook, adds target refinement loss, and constructs the MUSE multi-target reasoning segmentation benchmark.

  • Results

    PixelLM achieves state-of-the-art performance across MUSE, single-referring, and multi-referring segmentation benchmarks while reducing computational costs by up to 50%.

  • Takeaways & Limitations

    PixelLM provides an efficient LMM framework for generating high-quality masks across pixel-level reasoning and understanding tasks.

Abstract

from arXiv · show

While large multimodal models (LMMs) have achieved remarkable progress, generating pixel-level masks for image reasoning tasks involving multiple open-world targets remains a challenge. To bridge this gap, we introduce PixelLM, an effective and efficient LMM for pixel-level reasoning and understanding. Central to PixelLM is a novel, lightweight pixel decoder and a comprehensive segmentation codebook. The decoder efficiently produces masks from the hidden embeddings of the codebook tokens, which encode detailed target-relevant information. With this design, PixelLM harmonizes with the structure of popular LMMs and avoids the need for additional costly segmentation models. Furthermore, we propose a target refinement loss to enhance the model's ability to differentiate between multiple targets, leading to substantially improved mask quality. To advance research in this area, we construct MUSE, a high-quality multi-target reasoning segmentation benchmark. PixelLM excels across various pixel-level image reasoning and understanding tasks, outperforming well-established methods in multiple benchmarks, including MUSE, single- and multi-referring segmentation. Comprehensive ablations confirm the efficacy of each proposed component. All code, models, and datasets will be publicly available.

1. Introduction

PixelLM addresses the limited pixel-level response capabilities of multimodal models and the inability of prior reasoning segmentation methods to handle multiple targets without costly external segmentation models. It combines a pixel decoder, segmentation codebook, target refinement loss, and MUSE benchmark to improve efficient multi-target reasoning segmentation.

  • PixelLM targets pixel-level reasoning for fine-grained tasks where most multimodal models primarily produce textual descriptions.The gap affects applications including image editing, autonomous driving, and robotics.
  • Prior reasoning segmentation methods struggle with multiple target objects and depend on pre-trained external segmentation models.This dependence adds computational demands and ties performance to the external segmentation model.
  • PixelLM combines a lightweight pixel decoder, segmentation codebook, and target refinement loss to generate high-quality masks for arbitrary open-set targets.Codebook tokens encode target-relevant information, while the decoder uses their hidden embeddings with image features; the refinement loss improves differentiation between multiple targets.
  • MUSE provides a multi-target reasoning segmentation dataset with 246k question-answer pairs covering 0.9 million instances.The dataset was curated with a GPT-4V-aided pipeline and supports model training and evaluation.
  • PixelLM achieves state-of-the-art performance across MUSE, single-referring, and multi-referring segmentation benchmarks while reducing computational costs by up to 50%.The comparison is against baseline models, including methods that rely on external segmentation models.

2. Related Work

Related work extends multimodal models with language understanding and increasingly fine-grained visual grounding, but pixel-wise mask generation remains distinct from region grounding. PixelLM’s architecture is presented as a streamlined combination of a vision encoder, language model, pixel decoder, and segmentation codebook.

  • Earlier multimodal models commonly use smaller language models or train from scratch with contrastive and generative objectives.Their limited language understanding can hinder tasks requiring extensive world knowledge and reasoning.
  • LLM-augmented multimodal models typically integrate adapters to align visual and textual representations within language models.Examples include BLIP-2, Flamingo, MiniGPT-4, LLaVA, InstructBLIP, InternGPT, and Qwen-VL.
  • Fine-grained grounding methods encode image-region coordinates as tokens, but generally do not generate pixel-wise masks.LISA addresses segmentation by integrating SAM with an LLM for complex instruction reasoning.

3. Method

PixelLM equips an LMM with a segmentation codebook and lightweight pixel decoder that transform target-aware token embeddings and multi-scale image features into masks. Multiple tokens, sequential multi-scale decoding, target refinement loss, and MUSE support reasoning over varied numbers of open-set targets.

  • Framework overview: PixelLM combines a pre-trained CLIP-ViT encoder, large language model, lightweight pixel decoder, and segmentation codebook to produce interleaved descriptions and masks.The model processes image and query text while preserving the structure of popular LMMs.
  • Segmentation codebook: The segmentation codebook groups learnable tokens by visual scale, encoding target-relevant semantic and geometric information for mask generation.The codebook integrates diverse visual information across multiple semantic scales.
  • Segmentation codebook: Multiple tokens per scale provide complementary target information, improving mask generation when one token cannot capture multiple targets or complex semantics.Token fusion projects grouped hidden states into a unified representation before decoding.
  • Pixel decoder: The pixel decoder combines codebook embeddings with multi-scale image features, sequentially producing score maps that guide attention toward relevant regions at finer scales.It avoids requiring an additional costly segmentation model such as SAM.
  • Pixel decoder: The final mask aggregates predictions from all scales using learned weighting factors γ with |γ| = 1.Each decoder block corresponds to a distinct image-feature and codebook scale.
  • Training objectives: Target refinement loss focuses on unclear pixels where multiple targets overlap, helping the model distinguish different targets.The loss addresses increased confusion as the number of targets grows.
  • Dataset construction: MUSE addresses missing training data with open-set concepts, detailed descriptions, complex multi-target questions, and instance-level mask annotations.It contains 910k instance masks and 246k question-answer pairs averaging 3.7 targets per answer.
  • Evaluation: Evaluation matches predicted and ground-truth masks by IoU, obtains GPT scores for prediction quality, and multiplies them to calculate gIoU and cIoU.Unassigned predictions receive a score of 0.

5. Experiment

PixelLM is evaluated on multi-target reasoning, multi-referring, and conventional referring segmentation benchmarks, with ablations examining efficiency and key components. It consistently outperforms LISA-based baselines while using fewer computational resources, and MUSE training improves pixel reasoning.

  • Benchmarks and baselines: PixelLM is evaluated on MUSE, multi-referring segmentation, and conventional refCOCO referring segmentation benchmarks.The benchmarks vary from multiple targets to single-object segmentation.
  • Benchmark results: PixelLM consistently outperforms LISAaug and LISA-Llama2-13Baug across almost all evaluated metrics under matched training data and LMM sizes.The comparison uses equivalent training data and language-model sizes.
  • Efficiency: 50% and 35% reductions in TFlops are achieved by PixelLM at the 7B and 13B model sizes, respectively, compared to LISA.PixelLM-Llama2-13B uses 6.65 TFlops while scaling beyond the 7B model.
  • Benchmark results: PixelLM achieves the highest scores on most metrics in conventional referring segmentation, particularly on the more challenging refCOCO+/refCOCOg datasets.This extends the reported gains beyond multi-target benchmarks to single-target segmentation.
  • Ablation study: Adding one extra feature scale yields notable gains, while additional scales provide diminishing returns; token fusion increases cIoU by up to 2.4% on MUSE val and 2.2% on refCOCOg.The ablation also reports benefits from a larger laion-CLIP-H.
  • Ablation study: 0.8% gIoU and 0.6% cIoU improvements on MUSE val come from target refinement loss, while combining it with token fusion yields up to a 2.7% cIoU improvement.The target refinement loss does not affect refCOCOg performance.
  • Dataset effectiveness: MUSE training improves LISA on ReasonSeg val from 52.9 to 58.0 gIoU and from 54.0 to 59.1 cIoU.The result supports MUSE’s effectiveness for enhancing pixel reasoning on a single-object reasoning dataset.

6. Conclusion

PixelLM produces high-quality masks for diverse pixel-level image reasoning tasks and achieves promising results across various benchmarks. The study also introduces a multi-target reasoning segmentation benchmark to support further research.

  • PixelLM produces high-quality masks for a variety of pixel-level image reasoning tasks.
  • The study constructs a comprehensive multi-target reasoning segmentation benchmark to bolster research in this area.
  • PixelLM achieves promising results across various benchmarks through extensive experiments.

Supplementary Material

The supplementary material documents training and decoder details, evaluates MUSE, and provides additional analyses of the segmentation codebook, decoder, and multi-referring dataset.

  • The supplementary material details the training configuration, decoder structure, FLOPs calculation, and MUSE evaluation process.
  • Additional experiments analyze the segmentation codebook and decoder.
  • The material provides a more comprehensive analysis of MUSE and the multi-referring segmentation dataset.

A. Implementation Details

The implementation details describe training settings, a multi-scale pixel decoder, and an evaluation procedure that matches predicted masks to ground truth before scoring text-mask alignment and mask quality.

  • Pixel decoder: The pixel decoder uses attention at each scale, cross-scale feature modulation, and mask fusion to obtain the final result.The decoder processes multi-scale image features and segmentation tokens before combining masks from all scales.
  • Evaluation: Predictions and ground-truth objects are padded to a common size and matched by minimizing pairwise costs over permutations.The matching cost combines binary cross-entropy and Dice losses, and the optimal assignment is computed with the Hungarian algorithm.
  • Evaluation: The evaluation modifies generated responses by replacing segmentation-token sequences with GPT-generated descriptions of matched ground-truth objects.A GPT-3.5 prompt assigns each prediction a score, while empty predictions receive score 0.
  • Evaluation: Mask quality is evaluated after text-mask alignment using the final intersection-over-union of each prediction.The supplementary procedure then derives image-level IoU-based metrics, including gIoU and cIoU, for referring segmentation.
  • Implementation: The decoder implementation takes multi-scale image features and segmentation tokens, updates features across scales, and appends each scale's mask before fusion.The pseudocode defines learnable scale embeddings, output embeddings, positional embeddings, and mask weighting factors.

B. More Ablative Experiments.

Ablation studies examine whether segmentation tokens should be shared across scales and which vision-transformer layers should supply multi-scale features. Dedicated tokens for each scale perform better than sharing tokens across scales.

  • Multi-scale tokens sharing: Dedicated segmentation tokens for each feature scale yield better performance than sharing the same tokens across all scales.The token-sharing ablation reduces the codebook from N × L tokens to N shared tokens across L scales.

C.1. Data Filtering

The filtering process addresses GPT-4V-generated data failures, especially ambiguous questions and incomplete answers. The appendix also documents evaluation and dataset-statistics visualizations.

  • Filtering failures: GPT-4V filtering removes vague questions that permit multiple interpretations.The paper illustrates this problem with an outing-related question whose meaning is underspecified.
  • Filtering failures: Filtering also targets answers that omit semantically equivalent instances.Such omissions can leave relevant objects unrepresented despite the question’s intended meaning.
  • Evaluation and statistics: The appendix includes a GPT-based evaluation example showing an original predicted answer and its modified version.The modified answer is presented as input to GPT.
  • Evaluation and statistics: Dataset-statistics figures report category counts and the distribution of target counts in questions.These statistics are presented as appendix visualizations.

C.2. Dataset Statistics

MUSE spans over 1,000 object categories and uses descriptions that extend beyond category names into detailed, context-specific instance information.

  • Dataset breadth: MUSE question-answer pairs cover over 1,000 categories drawn from objects in daily scenes.The dataset therefore spans a broad range of everyday visual concepts.
  • Description richness: Object descriptions can exceed 100 tokens and include appearance, attributes, and relationships with other objects.These descriptions are generated through the GPT-4V-based data-generation pipeline.

C.3. Multi-referring Segmentation

The multi-referring transformation selects multiple annotated targets and converts them into ordered segmentation questions with one prediction for each described object.

  • Target selection: Each image contributes one to three distinct target objects selected from its annotations.The selected objects form the basis of multi-referring training examples.
  • Question and response format: Questions list comma-separated object descriptors, and responses must preserve that order when assigning each object its segmentation token.Evaluation computes gIoU and cIoU for each corresponding prediction.

C.4. More Details about GPT-4 Generated Data.

The GPT-4-generated data comparison uses captions instead of direct visual input, but produces less diverse and less detailed questions than GPT-4V. Additional experiments examine MUSE, decoder, and token-design effects.

  • GPT-4 versus GPT-4V: 30,000 additional multi-target question-answer pairs compare GPT-4 with GPT-4V using detailed image captions for GPT-4.Because GPT-4 cannot directly perceive image content, captions compensate for the missing visual information.
  • GPT-4 versus GPT-4V: GPT-4 caption-based generation often yields simple referring sub-questions, caption-limited content, and less detailed object descriptions.Adding relationship annotations could address these issues but would increase data-generation complexity and burden.
  • Evaluation scale: 15k samples in the enlarged MUSE‡ test set versus 4.3k in MUSE provide a larger evaluation setting.PixelLM performs similarly on both datasets, supporting consistency of the MUSE results.
  • Ablations: The default hyperparameter setting yields the best ablation results, while the larger laion-CLIP-H vision encoder improves PixelLM across all metrics.The comparison uses the same CE and DICE loss weights as LISA and identical hyperparameters across datasets.
  • MUSE effects: Training with MUSE markedly improves LISA’s performance on the single-object ReasonSeg dataset.The experiment compares LISA trained with and without MUSE data.
  • Token-design ablation: Predicting discrete coordinate tokens and 128 polygon points performs worse than PixelLM across all datasets.The longer 128-token sequence is identified as a likely source of additional autoregressive errors, especially for multi-instance masks.
  • MUSE effects: Adding MUSEtext improves all LLaVA-Bench metrics, while full MUSE training also improves multimodal performance and reduces hallucinations.The reported comparison uses 7B models and distinguishes MUSEtext from training with segmentation masks and loss.
Loading 2312.02228v3…