Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Marks

1. Introduction

• Prior region methods leave a shared image-video solution open, while video approaches face frame-proportional text input, repeated region-feature extraction, and inconsistent references across changing frames.Source: §1 · pp. 1–2
• Token Mark embeds a shared target token in prompted visual regions and corresponding text, providing a consistent reference across frames with text-token count independent of video length.Region prompts can be boxes or masks.Source: §1 · p. 2, Figure 2 · p. 2
• The Temporal Region Guide Head classifies visual tokens by Token Mark using the initial-frame prompt, supporting consistent video understanding at inference without full tracklets or added cost.Source: §1 · p. 2
• RegVID-300k provides region-level video instruction data, with 98k videos, 214k regions, and 294k instruction samples curated from 10 public datasets.Source: §1 · p. 2
• Omni-RGPT achieves state-of-the-art performance on image-based VCR and video-based Causal-VidQA, with enhanced video captioning and strong image captioning and referring-expression comprehension.Source: §1 · p. 2

2.1. Vision-Language MLLMs

• Fine-grained, region-specific understanding in images and videos remains an open challenge, especially for interactive settings requiring localized visual interpretation.Recent vision-language models have advanced global visual comprehension.Source: §2.1 · p. 2

2.2. Region-level MLLMs

• Earlier region-level methods encode box coordinates as text, use RoI-aligned features, or apply visual markers; Omni-RGPT supports boxes and masks without off-the-shelf tracking.Prior video approaches include initial-frame box prompts and scene-graph encoding.Source: §2.2 · p. 3

3. Omni-RGPT

• Omni-RGPT processes image or video features as visual tokens and combines them with a text prompt in the language model for joint reasoning.Region prompts identify designated areas across the spatio-temporal input.Source: §3 · p. 3

3.1. Token Mark

• Token Mark uses sampled tokens matched to region prompts, injecting them into language embeddings and spatial visual features to connect region and text representations.The spatial marks are pooled to the visual-token shape, projected, and added to visual tokens.Source: §3.1 · p. 4
• A shared region representation is intended to prevent temporal drift, connect regions directly to language, and preserve global vision-language alignment.Without region prompts, the architecture functions identically to the base model.Source: §3.1 · p. 4

3.2. Temporal Region Guide Head

• The Temporal Region Guide Head classifies visual tokens by Token Mark during training, guiding region interpretation across frames from an initial-frame prompt without tracklets.Soft labels represent visual tokens containing multiple Token Marks or background; the head adds no inference latency.Source: §3.2 · p. 4

4. Region-level Video Instruction Dataset

• The dataset pipeline generates region-level captions with GPT4o, mitigates caption hallucinations, then creates region-level instruction samples.The dataset is presented as a response to the lack of an existing region-level video instruction dataset.Source: §4 · p. 5
• RegVID-300k comprises 98k unique videos, 214k tracklets or masklets, and 294k instructions drawn from 10 public datasets.Its region descriptions average about 60 words and include contextual and temporal information.Source: §4 · p. 5

4.1. GPT4o-Assisted Region-level Captions

• For caption generation, SoM-style indexed masks and region nouns are provided to GPT4o to elicit enriched captions with contextual and temporal details.The videos are paired with region masklets.Source: §4.1 · p. 5

4.2. Visual Hallucination Mitigation

• For datasets with only tracklets or a single bounding box, SAM or SAM2 generates masklets.Source: §4.2 · p. 5
• Hallucination mitigation decomposes captions into closed-ended questions, checks them against videos with MLLMs, and removes unverified content before regeneration.The authors identify visual hallucinations in synthetically generated detailed captions as a fidelity concern.Source: §4.2 · pp. 5–6

4.3. GPT-Assited Region-level Instruction Data

• Text-only GPT4 creates region-specific question-answer pairs from detailed captions, including descriptions, summaries, and general questions.The instructions cover contextual attributes such as color and position, as well as temporal aspects such as motion and actions.Source: §4.3 · p. 6

5.1. Implementation Details

• Image pretraining follows the RegionGPT [19] recipe, then joint image-video fine-tuning; video data use four uniformly sampled frames, with a single linear auxiliary classifier (α=0.05).The setup uses Llama-2 [61], CLIP-ViT-L [52], and a two-layer MLP projection; fine-tuning includes region-level video datasets and RegVID-300k.Source: §5.1 · p. 6

5.2. Quantitative Evaluation

• Omni-RGPT achieves state-of-the-art performance across Causal-VidQA subtasks, with particular gains in Prediction and answer-plus-reason accuracy.The comparison reports strong performance without complete tracklet inputs or complex relational encoding modules.Source: §5.2 · pp. 6–7
• On VCR, Omni-RGPT achieves state-of-the-art performance across all metrics, outperforming methods designed specifically for image tasks.The three accuracy measures cover answer selection, rationale selection given the answer, and joint answer-and-rationale selection.Source: Table 4 · p. 7, §5.2 · p. 7
• Video captioning is strong across datasets except Elysium, whose brief noun-like captions favor finetuned or image-specialized methods.The evaluation also includes Extended-Elysium detailed captions and BenSMOT captions averaging 40 words.Source: §5.2 · p. 7, Table 3 · p. 7
• Image captioning scores are comparable to image-specialist methods, while image referring expression comprehension shows a significant margin.Source: Table 5 · p. 7, §5.2 · pp. 7–8, Table 6 · p. 7

5.3. Visualization Analysis

• Temporal Region Guide Head heatmaps predict prompted regions in the first frame and corresponding regions in later frames without further prompts.The visualization assigns Token Marks to the deer and hand regions in the initial frame.Source: Figure 5 · p. 8, §5.3 · p. 8

5.4. Ablation Study

• Removing the auxiliary task weakens detailed video captions, while reducing input frames particularly harms Prediction and Counterfactual accuracy.Scene Description and Explanation show more modest accuracy changes between four-frame and single-frame inputs.Source: §5.4 · p. 8, Table 7 · p. 8, Table 8 · p. 8
• Reducing image-token resolution significantly diminishes region-level understanding in COCO region classification.Source: §5.4 · p. 8, Table 9 · p. 8

• Robust region-level video understanding requires dense spatial and temporal visual information.Source: §6 · p. 8
• Four-frame video input may not fully capture the complexity of diverse real-world scenarios.Source: §6 · p. 8
• Extending region-level understanding to long-form videos is identified as a promising direction for future research.Source: §6 · p. 8

• Omni-RGPT uses Token Mark to bridge language and spatio-temporal visual tokens at the region level.Source: §7 · p. 8
• A video-specific auxiliary task and RegVID-300k are introduced to support robust video region understanding and detailed region descriptions.Source: §7 · p. 8
• The model surpasses existing methods on VCR and Causal-VidQA and performs strongly on region-level captioning and referring expression comprehension.Source: §7 · p. 8

via Token Marks

• The supplementary material outlines implementation and dataset setups, extended quantitative results, qualitative examples, visualization analyses, dataset details, and ethical considerations.Source: via Token Marks · p. 13

8. Implementation Details

• Training proceeds from image pretraining to image-video joint fine-tuning, incorporating RegVID-300k in the latter stage.Image pretraining uses paired image-text and image-region datasets; joint fine-tuning also uses RegionGPT's image-region dataset.Source: §8 · p. 13
• Both training stages run for one epoch; fine-tuning freezes the vision encoder, and the complete process takes 24 hours on 8 nodes of 8×A100 GPUs.The batch size is 16 and the learning rate is 5 × 10^-5.Source: §8 · p. 13

9.1. VCR

• Without category names, VCR performance decreases slightly; box prompts marginally trail masks, while joint image-video weights remain competitive without VCR-specific fine-tuning.The authors relate category-free performance to region semantics learned through Token Mark’s direct region-language connection.Source: §9.1 · pp. 13–14

9.2. Effectiveness of RegVID-300k

• Removing RegVID-300k annotations degrades Causal-VidQA Description by −1.3 and Extended-Elysium METOR by −5.8.The authors attribute the larger Extended-Elysium drop to the dataset’s diverse, detailed descriptions.Source: Table 11 · p. 14, §9.2 · p. 14

9.3. Hallucination

• On POPE/MSCOCO, Random Accuracy is best (F1 trails InstructBLIP by −0.47); Popular Accuracy trails RegionGPT by −0.57 but F1 leads, while Adversarial trails by −4.0 Accuracy and −1.97 F1.The authors characterize overall performance as competitive with image-specialized counterparts.Source: §9.3 · p. 14

10. More Qualitative Results

• The visualizations use a jointly fine-tuned 7B model and VIPSeg mask proposals, although VIPSeg is not part of the training data.Source: §10 · p. 14

10.1. Video Region-level Understanding

• Video examples show brief captions drawing on details across frames, detailed captions tracking location and motion changes, and answers about object relations and locations over time.Source: §10.1 · p. 14

10.2. Image Region-level Understanding

• Image examples cover detailed captions, commonsense QA, and REC; the authors attribute REC’s discrimination of similar objects to pixel-level visual-token references rather than averaged RoI appearance.Source: §10.2 · p. 14

11. Visualization Analysis

• Elysium videos tend to have monotonous motion and a single dominant object, while their ground-truth captions are often noun-based; the authors refine its annotations as Extended-Elysium.Source: §11 · p. 15
• Omni-RGPT sometimes understands small objects unstably, and its frame-independent visual encoding without explicit temporal embeddings limits its ability to interpret object direction.Source: §11 · p. 15
• The section also analyzes Temporal Region Guide Head heatmaps in scenarios with occluded objects.Source: §11 · p. 15

12.1. Data Preparation

• The sources are diversified across public datasets with region annotations; MeViS target nouns are generated from referring sentences, and long ActivityNet-Entities videos are split at event segments.Source: §12.1 · p. 15
• RegVID-300k includes multiple regions per video, and its captions primarily range from 30 to 60 words.Source: Figure 6 · p. 15, §12.1 · p. 15

12.2. Data Visualization

• The visualized QA samples demonstrate automatically generated, diverse question-and-answer pairs about video regions.Source: §12.2 · p. 15

12.3. GPT4o-Assisted Region-level Captioning

• For GPT4o-assisted region captioning, tracklet-only datasets receive SAM masklets, while single-frame-box datasets receive SAM2-propagated masklets; 16 resized frames are sampled for caption generation.Source: §12.3 · p. 16

12.4. Visual Hallucination Mitigation

• Caption verification decomposes captions into questions, checks them with Qwen2-VL on eight video frames, and regenerates captions after removing contents answered “Definitely No.”Source: §12.4 · p. 16

12.5. Region-level Instruction Data Generation

• Instruction data comprises detailed descriptions, brief descriptions, and region-level conversations; prompts are diversified, and conversations cover varied aspects of each region’s captions.Source: §12.5 · p. 16

13. Ethics Concerns

• Omni-RGPT may inherit societal biases in its responses from Llama-2 weights pretrained on large-scale web text.The model is fine-tuned on publicly available image and video datasets and re-annotated data.Source: §13 · p. 16
• Qualitative examples show region-level video captioning and QA alongside image captioning and QA.The examples include detailed descriptions of objects and answers to visual commonsense questions.Source: Table 15 · p. 18, Table 16 · p. 19, Table 17 · p. 20
• Reported failure cases include unstable performance on small objects and limited understanding of object direction.One example misidentifies a black chair as a backpack; another describes a cat as climbing upward when it is moving downward.Source: Table 22 · p. 23