PixelRefer: A Unified Framework for Spatio-Temporal Object Referring with Arbitrary Granularity

1 Introduction

• Existing MLLMs emphasize holistic image- and video-level interpretation, leaving fine-grained object-centric understanding of localized regions less addressed.The paper identifies such understanding as relevant to applications including human-computer interaction, embodied AI, medical diagnostics, and remote sensing.Source: §1 · p. 1
• Prior region-level approaches have limitations including repeated image encoding for multiple regions, captioning-focused tasks, and coarse or temporally inconsistent video representations.The introduction also notes that task-specific architectures can undermine MLLMs’ general-purpose capabilities.Source: §1 · pp. 1–2
• PixelRefer unifies region-level perception and reasoning across images and videos, spanning object-level to scene-level understanding while preserving general-purpose MLLM capabilities.The framework builds on the principle that region representations should integrate modularly with a general-purpose MLLM backbone.Source: §1 · pp. 2–3, Figure 1 · p. 2
• PixelRefer consistently achieves state-of-the-art performance across image and video object-centric tasks despite using fewer instruction samples than prior advanced counterparts.The evaluation spans recognition, captioning, and reasoning tasks.Source: Figure 2 · p. 3, §1 · p. 3
• SAOT uses a unified pixel-level mask representation and adapts to object scale to produce compact, semantically rich tokens from free-form regions.It is designed to preserve spatial context and can be integrated into general-purpose MLLMs with minimal modifications.Source: §1 · p. 3
• PixelRefer-Lite retains competitive accuracy while substantially reducing runtime and memory through an Object-Only Framework that pre-fuses global context into object tokens.The design follows the finding that global visual tokens receive attention mainly in early layers, while object tokens remain active throughout the LLM.Source: §1 · p. 3

2.1 Multimodal Large Language Models

• Video MLLMs process frame-wise or clip-level features but remain limited in fine-grained spatial and temporal reasoning, especially for object-centric tasks.Their visual features are interleaved with text tokens and processed by LLMs to generate responses.Source: §2.1 · p. 4

2.2 Region-level Multimodal Large Language Models

• Region-level methods improve localized understanding, but video approaches can lack multi-object interactions and temporal coherence, while DAM and PAM remain largely captioning-focused.The section also notes that task-specific architectures can compromise MLLMs’ general-purpose nature.Source: §2.2 · p. 4

2.3 Benchmarks and Datasets for Region-level MLLMs

• Existing benchmarks and instruction datasets largely center on object-level captioning, leaving complex spatiotemporal reasoning and tasks such as multi-object relations and multi-turn QA less supported.Traditional language metrics may miss factual correctness or fine-grained semantic alignment; newer evaluations use semantic similarity, semantic IoU, or attribute-based scoring.Source: §2.3 · pp. 4–5

3.1 Vision-Object Framework

• The Vision-Object Framework has a vision encoder, object tokenizer, text tokenizer, and instruction-following LLM; it integrates global context, object semantics, and instructions.Figure 5 presents it alongside the Object-Only Framework as a complementary paradigm.Source: §3.1 · p. 5, Figure 5 · p. 6
• The framework encodes a video into scene tokens, derives object tokens from user-specified free-form masks across timestamps, and feeds visual, object, and language tokens to the LLM.With no specified regions, it degenerates to general visual understanding; images are treated as single-frame videos.Source: §3.1 · p. 5, §3.2 · p. 5

3.2 Preliminary Analyses

• Across LLM layers, answer tokens attend more strongly to object tokens than to global visual tokens, making object tokens the primary semantic anchor for question answering.The analysis treats images as single-frame videos.Source: §3.2 · p. 5
• Attention spans object and global image tokens broadly in early layers, then shifts toward object tokens in deeper layers.This progression is described as early visual integration followed by selective retention of task-relevant object semantics.Source: §3.2 · p. 6

4.1 Overview

• PixelRefer combines global visual and object-level tokens; PixelRefer-Lite pre-integrates global context into object tokens and retains only those tokens for LLM decoding.The Lite design reduces computational cost while preserving critical semantic content.Source: §4.1 · p. 6, §4.3 · p. 9

4.2.1 Scale-Adaptive Object Tokenizer

• SAOT dynamically scales regions, extracts features from padded crops, masks object-specific features, encodes relative position, and aggregates redundant tokens.Small regions are upscaled to retain detail, while large regions are downscaled to reduce redundancy; k-means clustering preserves representative mean embeddings.Source: §4.2.1 · pp. 7–8

4.3 PixelRefer-Lite

• The two framework variants reflect attention patterns: global-token attention concentrates in early LLM layers, while object tokens remain active throughout.This analysis motivates transferring global context into object tokens before decoding in PixelRefer-Lite.Source: §4.1 · p. 6, §4.3 · pp. 8–9
• The Object-Centric Infusion module progressively injects local context and then global scene context into object tokens before LLM processing.The two cross-attention steps are described as preserving local appearance while adding broader scene semantics.Source: §4.3 · p. 9
• For video, PixelRefer-Lite processes temporally ordered object tokens, prepends timestamps, and independently applies OCI to each object mask.SAM 2 generates segmentation masks on sampled video frames.Source: §4.3 · pp. 9–10

5 Dataset

• The dataset collection addresses the limits of single-object captioning by organizing open-source image and video data into foundational perception and visual instruction tuning.The paper describes the resulting collection as a knowledge foundation for fine-tuning.Source: §5 · p. 10, §5.1 · p. 10

5.1.1 Foundational Object Perception Data

• Foundational Object Perception totals 1.4M samples across region recognition, detailed image captions, and detailed video captions.Its sources combine object, part, and temporal annotations with diverse image- and video-caption resources.Source: §5.1.1 · pp. 10–11

5.1.2 Visual Instruction Tuning Data

• Visual Instruction Tuning totals 0.8M samples and combines image-region QA, video-object QA, regional captions, and general visual QA.The sources span localized reasoning, open-ended queries, and caption-style supervision.Source: §5.1.2 · p. 11
• The collection includes 75K VideoRefer-QA samples for question–answer supervision over dynamic objects and their temporal interactions.The paper says these samples were constructed using a multi-agent pipeline.Source: §5.1.2 · p. 11

6.1 Experiment Setup

• The models use VideoLLaMA 3 and a two-stage training strategy: foundational object perception followed by visual instruction tuning.The main experiments use 32 object tokens; the setup reports training for one epoch per stage.Source: §6.1 · p. 11

6.2.1 Image-level Benchmarks

• PixelRefer reports leading image-level results across category recognition, captioning, and reasoning benchmarks.On PACO, PixelRefer-7B reaches 91.7% SSim and 85.3% SIoU, exceeding prior best results by 4.3% and 10.4%; it also reports gains on captioning and Ferret-Bench reasoning.Source: Table 1 · p. 12, §6.2.1 · p. 13

6.2.2 Video-level Benchmarks

• On VideoRefer-BenchD, PixelRefer leads regional-temporal understanding, with single-frame scores of 4.70 SC, 3.59 AD, and 3.39 TD, averaging +0.02 over multi-frame DAM-8B.The comparison uses single-frame input for PixelRefer and multi-frame input with denser object masks for DAM-8B.Source: §6.2.2 · pp. 13–14, Table 2 · p. 14
• PixelRefer scores 79.4% on VideoRefer-BenchQ, 8.1% above GPT-4o, and outperforms DAM-8B across all five reported HC-STVG metrics.PixelRefer-7B reaches 21.1 METEOR (+0.1), 97.4 CIDER (+6.1), 20.1 BLEU@4 (+0.3), 46.1 ROUGE-L (+0.2), and 32.5 SPICE (+1.1) on HC-STVG.Source: Table 3 · p. 14, §6.2.2 · pp. 14–15, Table 4 · p. 14

6.3 Efficiency Analysis

• With video inputs, PixelRefer-2B uses 11.15T FLOPs and 24.6GB memory, while PixelRefer-2B-Lite uses 0.11T FLOPs and 5.1GB.The paper reports similar reductions for larger models and image inputs, and reduced inference time and memory relative to DAM and PAM, particularly for video.Source: Table 5 · p. 14, §6.3 · p. 15, Table 6 · p. 15

6.4 Ablation Study

• Increasing object tokens beyond eight largely plateaus in Vision-Object, while Object-Only improves consistently; adding local and global context together gives the highest reported ablation scores.With 16 tokens, different object tokens attend to distinct object regions; combining local and global attention yields +4.4% on LVIS, +1.7% on DLC-Bench, +0.16 on VideoRefer-D, and +21.7% on HC-STVG.Source: Figure 10 · p. 15, Table 7 · p. 15, Table 9 · p. 15
• Adding image and video captioning data improves captioning, Region QA improves QA results, and General QA strengthens QA without impairing other tasks.Training begins with region-recognition data, on which the model scores 89.6% on LVIS, but initially struggles with detailed description and QA tasks.Source: Table 10 · p. 16, §6.4 · p. 17
• SAOT outperforms Mask Pooling by 6.8% on LVIS, 8.6% on DLC-Bench, and 1.4% on HC-STVG; gains are larger for small regions.For regions under 2000 pixels, improvements are 15.6% on LVIS and 9.6% on DLC-Bench; removing expansion lowers results by 5.2% on LVIS and 0.7% on DLC-Bench, while positional embeddings improve image and video benchmarks.Source: Table 8 · p. 15, Figure 12 · p. 16, §6.4 · pp. 16–17

• PixelRefer supports object-centric understanding across images and videos at arbitrary granularity.The framework is designed for fine-grained spatiotemporal understanding of user-specified regions.Source: §7 · p. 18
• SAOT generates compact, semantically rich representations from free-form regions, and PixelRefer-2.2M supplies object-centric instruction data for training.Source: §7 · p. 18
• PixelRefer-Lite pre-fuses global context into object tokens and reduces computational cost while maintaining high semantic fidelity.The conclusion reports state-of-the-art performance for PixelRefer with fewer training samples and comparable accuracy with efficiency gains for Lite.Source: §7 · p. 18