PaperlayerView exact revision on arXiv

Source-linked AI summary

UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoning

Ye Liu, Zongyang Ma, Junfu Pu, Zhongang Qi, Yang Wu, Ying Shan, Chang Wen Chen

arXiv:2509.18094v4cs.CVcs.AI

TL;DR

Holistic LMMs remain limited in fine-grained pixel-level interaction and do not integrate referring and segmentation into visual reasoning. UniPixel addresses this with visual-prompt processing, an object memory bank, and mask-conditioned reasoning, achieving strong results across 10 benchmarks and establishing a baseline for PixelQA.

  • Problem

    Holistic LMMs lack fine-grained pixel-level alignment and existing methods generally perform referring or segmentation independently rather than integrating them into visual reasoning.

  • Method

    UniPixel encodes point, box, and mask prompts, generates spatial-temporal masks, stores object information in an object memory bank, and conditions reasoning on it.

  • Results

    UniPixel achieves state-of-the-art performance across 10 public benchmarks and establishes a strong baseline for PixelQA; its 7B model reaches 64.0 J &F on ReVOS.

  • Takeaways & Limitations

    The unified representation jointly enhances object referring and segmentation while supporting pixel-level reasoning across images and videos.

  • Takeaways & Limitations

    Training did not include additional pixel-level tasks such as grounded caption generation, and the mask decoder propagates the first-frame mask rather than selecting the best frame.

Abstract

from arXiv · show

Recent advances in Large Multi-modal Models (LMMs) have demonstrated their remarkable success as general-purpose multi-modal assistants, with particular focuses on holistic image- and video-language understanding. Conversely, less attention has been given to scaling fine-grained pixel-level understanding capabilities, where the models are expected to realize pixel-level alignment between visual signals and language semantics. Some previous studies have applied LMMs to related tasks such as region-level captioning and referring expression segmentation. However, these models are limited to performing either referring or segmentation tasks independently and fail to integrate these fine-grained perception capabilities into visual reasoning. To bridge this gap, we propose UniPixel, a large multi-modal model capable of flexibly comprehending visual prompt inputs and generating mask-grounded responses. Our model distinguishes itself by seamlessly integrating pixel-level perception with general visual understanding capabilities. Specifically, UniPixel processes visual prompts and generates relevant masks on demand, and performs subsequent reasoning conditioning on these intermediate pointers during inference, thereby enabling fine-grained pixel-level reasoning. The effectiveness of our approach has been verified on 10 benchmarks across a diverse set of tasks, including pixel-level referring/segmentation and object-centric understanding in images/videos. A novel PixelQA task that jointly requires referring, segmentation, and question answering is also designed to verify the flexibility of our method.

1 Introduction

UniPixel addresses the limited fine-grained interaction supported by holistic vision-language models by unifying visual prompting, referring, segmentation, and pixel-level reasoning. It uses an object memory bank to generate mask-grounded responses and reports strong results across diverse benchmarks, including a new PixelQA task.

  • 1 Introduction: Holistic vision-language models mainly support text interactions and lack intuitive visual references or grounding of responses in key regions.
  • 1 Introduction: UniPixel unifies object referring and segmentation with pixel-level reasoning through an object memory bank that stores and injects object-centric spatial-temporal information.The model updates this memory on demand and conditions responses on the resulting fine-grained object information.
  • 1 Introduction: UniPixel accepts points, boxes, and masks as visual prompts and generates mask-grounded responses within a single model.
  • 1 Introduction: 10 public benchmarks across 8 image/video referring and segmentation tasks show state-of-the-art performance, including 62.1 J &F on ReVOS and 72.8% Acc on VideoRefer-BenchQ from the 3B model.The 3B model surpasses strong counterparts with 7B–13B parameters on the highlighted video tasks.
  • 1 Introduction: PixelQA jointly evaluates object-centric referring, segmentation, and question answering in videos, and UniPixel establishes a strong baseline for this setting.

2 Related Work

Prior work enhanced LMMs with visual referring and segmentation, but existing unified approaches remained image-only or tool-based. UniPixel presents an architecture that encodes visual prompts, predicts masks, and feeds object memory into pixel-level reasoning.

  • Large Multi-modal Models: Large multi-modal models use visual-language alignment and instruction tuning to support holistic tasks such as captioning and general question answering.
  • Visual Referring and Segmentation: Existing referring-and-segmentation approaches either support only images or rely on sub-optimal tool-based pipelines, whereas UniPixel unifies the capabilities end to end.
  • Visual Referring and Segmentation: UniPixel encodes videos, questions, and visual prompts, predicts spatial-temporal masks, stores them in an object memory bank, and injects them into subsequent reasoning.

3 Method

UniPixel unifies visual prompting, object memory, and mask decoding to support pixel-level reasoning over images and videos. Its method encodes spatial-temporal prompts, stores object masks for later injection, and trains language and mask prediction jointly.

  • Problem Formulation: UniPixel defines pixel-level reasoning as mapping an image or video, text, and optional point, box, or mask prompts to textual responses with grounded spatial-temporal masks.The number of input prompts and output masks may differ, and both can be zero for ordinary visual understanding.
  • Prompt Encoder: The prompt encoder converts point, box, and mask inputs into tokens that the language model can process, encoding spatial coordinates, frame indices, and dense binary masks.Point and box prompts include frame information, while mask prompts match the encoded target frame’s shape.
  • Prompt Encoder: Figure 4 illustrates joint positional and temporal encoding for point and box prompts, including coordinates and frame indices in their compact representations.Point prompts use (X1Y1T), while box prompts use (X1Y1X2Y2T).
  • Object Memory Bank: The object memory bank stores object IDs with spatial-temporal masks, enabling sparse visual prompts to be connected to dense object information across multi-round interactions.Memory pre-filling predicts masks for referred objects and saves object-mask pairs; memory injection pools frame-level masks into features that replace <MEM> tokens.
  • Mask Decoder and Training: UniPixel uses memory-enhanced object representations to separate regional understanding from mask prediction, allowing referring and segmentation data to support both capabilities.The mask decoder uses <SEG> hidden states to predict masks, while training combines language modeling with focal, dice, IoU, and objectness losses in a three-stage alignment recipe.

4 Experiments

UniPixel is evaluated on basic referring and segmentation, video reasoning, PixelQA, and architectural ablations. It achieves strong performance across diverse image/video tasks, while ablations support benefits from task unification and the object memory bank.

  • Q1: Comparison with State-of-the-Arts on Referring and Segmentation Tasks: UniPixel consistently achieves the best performance on MeViS, Ref-YouTube-VOS, and Ref-DAVIS17, with its MeViS advantage reaching around 3.5% over GLUS-7B and 17% over VideoGLaMM-3.8B.The method also performs better on Ref-SAV despite long descriptions, large motion, camera motion, and heavy occlusion.
  • Q1: Comparison with State-of-the-Arts on Referring and Segmentation Tasks: UniPixel significantly outperforms the GroundMoRe baseline on motion-grounded video reasoning, which requires joint spatial and temporal grounding.The comparison mainly uses MoRA fine-tuned on GroundMoRe, whereas other methods are evaluated zero-shot.
  • Q1: Comparison with State-of-the-Arts on Referring and Segmentation Tasks: UniPixel achieves competitive referring expression comprehension by inferring bounding boxes from its predicted masks.The task measures accuracy at IoU ≥ 0.5, and the reported performance benefits from high-quality mask prediction.
  • Q2: Pixel-Level Video Question Answering (PixelQA): PixelQA replaces mask prompts with point or box prompts and requires models to identify the target object, answer the question, and output corresponding masks.The task also evaluates mixed prompt types, and the reported results support memory pre-filling and injection for reasoning.
  • Q3: Key Ablation Studies: Jointly training referring and segmentation improves both tasks, while adding memory pre-filling as an auxiliary task yields further gains.The ablation is reported on PixelQA using UniPixel-3B.
  • Q3: Key Ablation Studies: Auxiliary segmentation and masked-pooled features improve regional understanding, while decoupling them through the object memory bank further boosts performance.The object-memory ablation compares single-object tokens, added segmentation tokens, and appended masked-pooled visual tokens.

5 Conclusion

UniPixel supports flexible pixel-level visual reasoning by unifying referred and segmented objects through an object memory bank. Experiments across diverse tasks, including PixelQA, show the significance of this unified approach.

  • UniPixel supports flexible pixel-level visual reasoning by unifying referred and segmented object representations through an object memory bank.The model is designed to integrate object referring and segmentation within one framework.
  • The proposed unification jointly enhances object referring and segmentation performance across diverse pixel-level understanding tasks.
  • The appendix contains additional implementation details, training information, experiments, ablations, visualizations, and discussions.

A.1 Implementation Details

UniPixel uses 3B and 7B Qwen2.5-VL base models with SAM 2.1 mask decoders and a 256-dimensional prompt encoder. Video inputs are sampled sparsely to reduce memory use and accelerate training.

  • UniPixel instantiates 3B and 7B Qwen2.5-VL base models with pre-trained SAM 2.1 mask decoders using Hiera Base+ backbones.
  • The M→L projector inherits Qwen2.5-VL’s V→L projector weights, while the prompt encoder has hidden size 256.
  • Video training samples 8 frames per video, resizing each frame to 3162 ∼4482 pixels and representing it with 128 ∼256 tokens.This setup reduces GPU memory use and accelerates training.

A.2 Training Recipe

UniPixel is trained in three stages that progressively support visual-prompt comprehension, regional captioning, and object-centric understanding. Its evaluation covers 9 fine-grained image/video tasks spanning referring, segmentation, reasoning, description, and question answering.

  • The first training stage pre-trains a sparse prompt encoder on Inst-IT and VideoRefer caption samples for visual-prompt comprehension and regional captioning.Each sample uses either a random point inside the ground-truth mask or an augmented box.
  • The training-dataset distribution distinguishes object referring, object segmentation, regional understanding, and memory pre-filling data categories.
  • UniPixel is evaluated on 9 fine-grained image/video understanding tasks covering referring, segmentation, motion-grounded reasoning, description, and question answering.

B.2 Evaluation Metrics

Evaluation uses task-specific metrics for video and image segmentation, referred video description and question answering, and referring expression comprehension. Additional tables address general VideoQA comparison and multi-stage training effectiveness.

  • Video segmentation uses J &F, combining region similarity J with contour accuracy F, while image segmentation uses cIoU and gIoU.
  • Referred video description and question answering are evaluated with GPT-4o scores and MCQ accuracy, respectively.
  • The VideoQA comparison table notes that UniPixel is the only listed model supporting pixel-level referring and segmentation.
  • The multi-stage training ablation reports that the three-stage recipe achieves optimal performance.
  • Referring expression comprehension uses mean accuracy, counting a predicted box as correct when its IoU with ground truth is at least 0.5.

B.3 More Experimental Results

UniPixel is evaluated for general video understanding on MVBench alongside existing methods. It balances holistic video capabilities with referring and segmentation through joint training on holistic- and pixel-level data.

  • UniPixel is the only method in the MVBench comparison that supports both referring and segmentation.
  • Joint training on holistic- and pixel-level data preserves strong general video understanding alongside pixel-level capabilities.

B.4 Ablation Studies

The ablations show that staged alignment, two hidden tokens per object, and progressively expanded pixel-level training data support UniPixel’s performance. Qualitative outputs cover diverse prompt and reasoning types.

  • Pre-aligning the sparse prompt encoder and L→M projector further boosts referring and segmentation performance over direct large-scale training.Direct training is described as sub-optimal because prompt encoder, LLM, and mask decoder representations are unaligned.
  • Two hidden tokens per object are selected because one loses object information, while four or eight provide negligible additional gains.The final model therefore uses 2 hidden tokens per object.
  • Gradually adding regional, segmentation, memory-prefilling, and holistic video data improves the combined training strategy’s performance.The cited table caption states that adding more pixel-level data brings performance gains.
  • UniPixel handles flexible visual prompts, implicit and long queries, and motion-grounded questions across pixel-level understanding tasks.

C.1 Limitations & Future Work

The authors identify resource-limited training scope and a first-frame mask-propagation design as limitations and future-work targets. Qualitative examples span multiple image and video pixel-level tasks.

  • Limited computing resources prevented scaling training to additional pixel-level tasks such as grounded caption generation for images or videos.The authors suggest that adding such data may bring further performance gains.
  • The mask decoder predicts the first mask on the first frame and propagates it forward, while future work may predict on the best frame and propagate in both directions.
  • Qualitative examples cover PixelQA, referring and segmentation prompts, video questions, referred descriptions, and other image- and video-understanding tasks.
Loading 2509.18094v4…