PixelLM: Pixel Reasoning with Large Multimodal Model

• PixelLM targets pixel-level reasoning for fine-grained tasks where most multimodal models primarily produce textual descriptions.The gap affects applications including image editing, autonomous driving, and robotics.Source: §1 · p. 2
• Prior reasoning segmentation methods struggle with multiple target objects and depend on pre-trained external segmentation models.This dependence adds computational demands and ties performance to the external segmentation model.Source: §1 · p. 2
• PixelLM combines a lightweight pixel decoder, segmentation codebook, and target refinement loss to generate high-quality masks for arbitrary open-set targets.Codebook tokens encode target-relevant information, while the decoder uses their hidden embeddings with image features; the refinement loss improves differentiation between multiple targets.Source: §1 · p. 2
• MUSE provides a multi-target reasoning segmentation dataset with 246k question-answer pairs covering 0.9 million instances.The dataset was curated with a GPT-4V-aided pipeline and supports model training and evaluation.Source: §1 · p. 2
• PixelLM achieves state-of-the-art performance across MUSE, single-referring, and multi-referring segmentation benchmarks while reducing computational costs by up to 50%.The comparison is against baseline models, including methods that rely on external segmentation models.Source: §1 · p. 2

• Earlier multimodal models commonly use smaller language models or train from scratch with contrastive and generative objectives.Their limited language understanding can hinder tasks requiring extensive world knowledge and reasoning.Source: §2.1 · p. 2
• LLM-augmented multimodal models typically integrate adapters to align visual and textual representations within language models.Examples include BLIP-2, Flamingo, MiniGPT-4, LLaVA, InstructBLIP, InternGPT, and Qwen-VL.Source: §2.1 · p. 2
• Fine-grained grounding methods encode image-region coordinates as tokens, but generally do not generate pixel-wise masks.LISA addresses segmentation by integrating SAM with an LLM for complex instruction reasoning.Source: §2.2 · pp. 2–3

Framework overview

• PixelLM combines a pre-trained CLIP-ViT encoder, large language model, lightweight pixel decoder, and segmentation codebook to produce interleaved descriptions and masks.The model processes image and query text while preserving the structure of popular LMMs.Source: §3.1 · p. 3, §3 · p. 3

Segmentation codebook

• The segmentation codebook groups learnable tokens by visual scale, encoding target-relevant semantic and geometric information for mask generation.The codebook integrates diverse visual information across multiple semantic scales.Source: §3.1 · p. 3
• Multiple tokens per scale provide complementary target information, improving mask generation when one token cannot capture multiple targets or complex semantics.Token fusion projects grouped hidden states into a unified representation before decoding.Source: §3.1 · p. 4, Figure 3 · p. 4

Pixel decoder

• The pixel decoder combines codebook embeddings with multi-scale image features, sequentially producing score maps that guide attention toward relevant regions at finer scales.It avoids requiring an additional costly segmentation model such as SAM.Source: §3.1 · p. 4
• The final mask aggregates predictions from all scales using learned weighting factors γ with |γ| = 1.Each decoder block corresponds to a distinct image-feature and codebook scale.Source: §3.1 · p. 4

Training objectives

• Target refinement loss focuses on unclear pixels where multiple targets overlap, helping the model distinguish different targets.The loss addresses increased confusion as the number of targets grows.Source: §3.2 · p. 4

Dataset construction

• MUSE addresses missing training data with open-set concepts, detailed descriptions, complex multi-target questions, and instance-level mask annotations.It contains 910k instance masks and 246k question-answer pairs averaging 3.7 targets per answer.Source: §3.2 · p. 5, §4.1 · p. 5

Evaluation

• Evaluation matches predicted and ground-truth masks by IoU, obtains GPT scores for prediction quality, and multiplies them to calculate gIoU and cIoU.Unassigned predictions receive a score of 0.Source: §4.3 · p. 6

Benchmarks and baselines

• PixelLM is evaluated on MUSE, multi-referring segmentation, and conventional refCOCO referring segmentation benchmarks.The benchmarks vary from multiple targets to single-object segmentation.Source: §5.2 · p. 6

Benchmark results

• PixelLM consistently outperforms LISAaug and LISA-Llama2-13Baug across almost all evaluated metrics under matched training data and LMM sizes.The comparison uses equivalent training data and language-model sizes.Source: §5.3 · p. 7, §5.4 · p. 7

Efficiency

• 50% and 35% reductions in TFlops are achieved by PixelLM at the 7B and 13B model sizes, respectively, compared to LISA.PixelLM-Llama2-13B uses 6.65 TFlops while scaling beyond the 7B model.Source: §5.3 · p. 7

Benchmark results

• PixelLM achieves the highest scores on most metrics in conventional referring segmentation, particularly on the more challenging refCOCO+/refCOCOg datasets.This extends the reported gains beyond multi-target benchmarks to single-target segmentation.Source: §5.4 · p. 7

Ablation study

• Adding one extra feature scale yields notable gains, while additional scales provide diminishing returns; token fusion increases cIoU by up to 2.4% on MUSE val and 2.2% on refCOCOg.The ablation also reports benefits from a larger laion-CLIP-H.Source: §5.5 · p. 7
• 0.8% gIoU and 0.6% cIoU improvements on MUSE val come from target refinement loss, while combining it with token fusion yields up to a 2.7% cIoU improvement.The target refinement loss does not affect refCOCOg performance.Source: §5.5 · p. 8

Dataset effectiveness

• MUSE training improves LISA on ReasonSeg val from 52.9 to 58.0 gIoU and from 54.0 to 59.1 cIoU.The result supports MUSE’s effectiveness for enhancing pixel reasoning on a single-object reasoning dataset.Source: §5.5 · p. 8

• PixelLM produces high-quality masks for a variety of pixel-level image reasoning tasks.Source: §6 · p. 8
• The study constructs a comprehensive multi-target reasoning segmentation benchmark to bolster research in this area.Source: §6 · p. 8
• PixelLM achieves promising results across various benchmarks through extensive experiments.Source: §6 · p. 8

• The supplementary material details the training configuration, decoder structure, FLOPs calculation, and MUSE evaluation process.Source: Supplementary Material · p. 11
• Additional experiments analyze the segmentation codebook and decoder.Source: Supplementary Material · p. 11
• The material provides a more comprehensive analysis of MUSE and the multi-referring segmentation dataset.Source: Supplementary Material · p. 11

Pixel decoder

• The pixel decoder uses attention at each scale, cross-scale feature modulation, and mask fusion to obtain the final result.The decoder processes multi-scale image features and segmentation tokens before combining masks from all scales.Source: A. Implementation Details · p. 11

Evaluation

• Predictions and ground-truth objects are padded to a common size and matched by minimizing pairwise costs over permutations.The matching cost combines binary cross-entropy and Dice losses, and the optimal assignment is computed with the Hungarian algorithm.Source: A. Implementation Details · p. 11
• The evaluation modifies generated responses by replacing segmentation-token sequences with GPT-generated descriptions of matched ground-truth objects.A GPT-3.5 prompt assigns each prediction a score, while empty predictions receive score 0.Source: A. Implementation Details · p. 11
• Mask quality is evaluated after text-mask alignment using the final intersection-over-union of each prediction.The supplementary procedure then derives image-level IoU-based metrics, including gIoU and cIoU, for referring segmentation.Source: A. Implementation Details · pp. 11–12

Implementation

• The decoder implementation takes multi-scale image features and segmentation tokens, updates features across scales, and appends each scale's mask before fusion.The pseudocode defines learnable scale embeddings, output embeddings, positional embeddings, and mask weighting factors.Source: A. Implementation Details · p. 12

Multi-scale tokens sharing

• Dedicated segmentation tokens for each feature scale yield better performance than sharing the same tokens across all scales.The token-sharing ablation reduces the codebook from N × L tokens to N shared tokens across L scales.Source: B. More Ablative Experiments · p. 12

Filtering failures

• GPT-4V filtering removes vague questions that permit multiple interpretations.The paper illustrates this problem with an outing-related question whose meaning is underspecified.Source: §C.1 · p. 12
• Filtering also targets answers that omit semantically equivalent instances.Such omissions can leave relevant objects unrepresented despite the question’s intended meaning.Source: §C.1 · p. 12

Evaluation and statistics

• The appendix includes a GPT-based evaluation example showing an original predicted answer and its modified version.The modified answer is presented as input to GPT.Source: Figure A.1 · p. 13
• Dataset-statistics figures report category counts and the distribution of target counts in questions.These statistics are presented as appendix visualizations.Source: Figure A.2 · p. 13, §C.1 · p. 13

Dataset breadth

• MUSE question-answer pairs cover over 1,000 categories drawn from objects in daily scenes.The dataset therefore spans a broad range of everyday visual concepts.Source: §C.2 · p. 13

Description richness

• Object descriptions can exceed 100 tokens and include appearance, attributes, and relationships with other objects.These descriptions are generated through the GPT-4V-based data-generation pipeline.Source: §C.2 · p. 13

Target selection

• Each image contributes one to three distinct target objects selected from its annotations.The selected objects form the basis of multi-referring training examples.Source: §C.3 · p. 13

Question and response format

• Questions list comma-separated object descriptors, and responses must preserve that order when assigning each object its segmentation token.Evaluation computes gIoU and cIoU for each corresponding prediction.Source: §C.3 · p. 14

GPT-4 versus GPT-4V

• 30,000 additional multi-target question-answer pairs compare GPT-4 with GPT-4V using detailed image captions for GPT-4.Because GPT-4 cannot directly perceive image content, captions compensate for the missing visual information.Source: §C.4 · p. 14
• GPT-4 caption-based generation often yields simple referring sub-questions, caption-limited content, and less detailed object descriptions.Adding relationship annotations could address these issues but would increase data-generation complexity and burden.Source: §C.4 · p. 14

Evaluation scale

• 15k samples in the enlarged MUSE‡ test set versus 4.3k in MUSE provide a larger evaluation setting.PixelLM performs similarly on both datasets, supporting consistency of the MUSE results.Source: §C.4 · p. 15

Ablations

• The default hyperparameter setting yields the best ablation results, while the larger laion-CLIP-H vision encoder improves PixelLM across all metrics.The comparison uses the same CE and DICE loss weights as LISA and identical hyperparameters across datasets.Source: §C.4 · p. 15

MUSE effects

• Training with MUSE markedly improves LISA’s performance on the single-object ReasonSeg dataset.The experiment compares LISA trained with and without MUSE data.Source: §C.4 · p. 15

Token-design ablation

• Predicting discrete coordinate tokens and 128 polygon points performs worse than PixelLM across all datasets.The longer 128-token sequence is identified as a likely source of additional autoregressive errors, especially for multi-instance masks.Source: §C.4 · p. 15

MUSE effects

• Adding MUSEtext improves all LLaVA-Bench metrics, while full MUSE training also improves multimodal performance and reduces hallucinations.The reported comparison uses 7B models and distinguishes MUSEtext from training with segmentation masks and loss.Source: §C.4 · p. 15