PaperlayerView exact revision on arXiv

Source-linked AI summary

OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding

Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji, Chen Change Loy, Shuicheng Yan

arXiv:2406.19389v2cs.CV

TL;DR

LLaVA-like models lack precise location outputs, while unifying image-, object-, and pixel-level tasks in one model remains an open question. OMG-LLaVA combines a visual encoder, decoder, and LLM for these tasks, and the authors report comparable results to previous combined works with fewer trainable parameters and computation costs.

  • Problem

    LLaVA-like models cannot output precise location information, leaving the question of how to unify image-, object-, and pixel-level tasks in one MLLM.

    Source: §1 · p. 2

  • Method

    OMG-LLaVA uses one visual encoder, one decoder, and one LLM, with text, pixel-centric, and object-centric tokens and perception-prior embedding for text and segmentation outputs.

    Source: §1 · p. 3, §3.1 · p. 5, §5 · p. 10

  • Results

    Across over 8 multimodal tasks, OMG-LLaVA achieves comparable results to previous combined works with much fewer trainable parameters and computation costs while preserving OMG-Seg's visual perception ability.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The authors suggest this meta-architecture may inspire MLLM designs that minimize model components while maximizing functionality.

    Source: §5 · p. 10

    OMG-LLaVA cannot perform part-level segmentation because OMG-Seg lacks multi-granularity segmentation capability.

    Source: §A.5 · p. 19

Abstract

from arXiv · show

Current universal segmentation methods demonstrate strong capabilities in pixel-level image and video understanding. However, they lack reasoning abilities and cannot be controlled via text instructions. In contrast, large vision-language multimodal models exhibit powerful vision-based conversation and reasoning capabilities but lack pixel-level understanding and have difficulty accepting visual prompts for flexible user interaction. This paper proposes OMG-LLaVA, a new and elegant framework combining powerful pixel-level vision understanding with reasoning abilities. It can accept various visual and text prompts for flexible user interaction. Specifically, we use a universal segmentation method as the visual encoder, integrating image information, perception priors, and visual prompts into visual tokens provided to the LLM. The LLM is responsible for understanding the user's text instructions and providing text responses and pixel-level segmentation results based on the visual information. We propose perception prior embedding to better integrate perception priors with image features. OMG-LLaVA achieves image-level, object-level, and pixel-level reasoning and understanding in a single model, matching or surpassing the performance of specialized methods on multiple benchmarks. Rather than using LLM to connect each specialist, our work aims at end-to-end training on one encoder, one decoder, and one LLM. The code and model have been released for further research.

1 Introduction

OMG-LLaVA combines image-, object-, and pixel-level reasoning and understanding in one model, using a universal segmentation encoder to support text and visual-prompt interaction. It adds perception-prior embedding and a unified instruction strategy, and reports performance across six datasets.

1 Introduction

  • LLaVA-like models [69] [70] lack precise localization; task-specific grounding models [133] [94] [47] may lose caption and visual-question-answering abilities, while multi-model agents [124] [47] [85] [76] incur high inference and parameter costs.The introduction identifies the absence of specific designs for task unification as another limitation of agent-based approaches.Source: §1 · p. 2
  • OMG-Seg serves as the universal perception model, with its visual encoder and decoder frozen to preserve pixel-level segmentation ability.The paper selects OMG-Seg for its simplicity and effectiveness across segmentation tasks.Source: §1 · p. 2
  • OMG-LLaVA unifies image-, object-, and pixel-level reasoning in one LLM, encoder, and decoder while retaining image understanding and visual-prompt interaction.It supports image-level captioning and conversation, visual-prompt-based conversation and region captions, as well as referring and reasoning segmentation and grounded conversation and generation.Source: §1 · pp. 2–3
  • Perception prior embedding converts object queries into object-centric visual tokens, while unified instructions map image, text, and visual prompts to text, segmentation tokens, masks, and labels.The model follows pretraining and instruction-tuning pipelines.Source: §1 · p. 3
  • The paper reports good-enough performance on six datasets spanning panoptic segmentation, referring segmentation, grounded conversation, and region captioning.The datasets are COCO panoptic segmentation, VIPSeg video panoptic segmentation, refCOCO, refCOCO+, refCOCOg referring expression segmentation, GranDf grounded conversation generation, and refCOCOg region captioning; experiments also assess the components and training strategy.Source: §1 · p. 3

2 Related Work

OMG-LLaVA is positioned as a unified model connecting multimodal instruction-following and reasoning with universal segmentation and visual prompting. The related work contrasts this goal with approaches that specialize in individual tasks or rely on multiple visual experts.

Multimodal Large Language Models

  • To the authors’ knowledge, OMG-LLaVA is the first model to unify segmentation, instruction tuning, and prompt-driven segmentation in one simple model.Earlier LLaVA extensions address visual grounding, detection, segmentation, or video analysis, but are trained for specific purposes.Source: §2 · p. 3
  • Recent MLLMs [3] [90] [69] [17] build instruction-tuning pipelines across multimodal benchmarks, while LLaVA [69] [104] treats visual features as tokens.Later work explores visual cues for LLaVA [124] and adds components for grounding, detection, segmentation, and video analysis [125] [86] [129] [24] [25] [64] [128] [81] [36] [47].Source: §2 · p. 3

Unified Segmentation Models

  • Universal segmentation research uses end-to-end mask classification and set prediction, with methods outperforming specialized models in image, video, and generalization segmentation tasks [43] [59].Shared-parameter models [39] [31] address multiple segmentation tasks; OMG-Seg unifies image, video, open-vocabulary, and interactive segmentation, but these methods lack MLLM-like interactive text and visual prompts.Source: §2 · p. 3

Language-driven Location and Segmentation

  • Language-driven work covers referring segmentation and localization [66] [42] [23] [133], fusion modules [29] [5] [75], reasoning and region-level captioning [106] [47] [85] [124] [79], and LLM-agent visual experts [28] [70].LISA [112] studies reasoning-based segmentation, while GLaMM [85] introduces region-level caption and segmentation tasks; OMG-LLaVA contrasts with these approaches through one visual encoder, one LLM, and one decoder.Source: §2 · pp. 3–4

Visual Prompts

  • Visual-prompting methods include learnable tokens, mask-visual modeling [27] [96], and prompting encoders for visual outputs [97] [99].OMG-LLaVA also supports visual prompts to interact with user inputs.Source: §2 · p. 4

3 Methodology

OMG-LLaVA frames image-, object-, and pixel-level understanding and reasoning as token-to-token generation, using a frozen universal perception module and an LLM to produce text and segmentation outputs. Its method combines object-aware visual prompting and perception priors with two-stage training.

3.1 Task Unification

  • OMG-LLaVA unifies image-, object-, and pixel-level tasks as token-to-token generation using text, pixel-centric, and object-centric visual tokens.The framework also supports mixed-level tasks, including grounded descriptions of specified objects.Source: §3.1 · p. 5

3.2 OMG-LLaVA Framework

  • The frozen universal perception module encodes images and visual prompts into visual tokens, while the LLM produces text responses and [SEG] outputs decoded into masks.Separate visual projectors map pixel- and object-centric tokens into the LLM’s text embedding space; instruction formulation uses <Image>, <Region>, and [SEG] tokens.Source: §3.2 · pp. 5, 7, Figure 3 · p. 5
  • The OMG decoder uses learnable object queries and visual-prompt queries to extract object-centric tokens, which can be decoded into masks and object categories.Attention masks derived from box coordinates or provided object masks preserve prompt information, enabling point, box, and mask prompts.Source: §3.2 · p. 6
  • Perception prior embedding combines image features with mask- and confidence-weighted object queries to provide the LLM with pixel- and object-centric visual tokens.The strategy addresses the difficulty of combining a frozen perception module with an LLM without fine-tuning the perception module.Source: §3.2 · pp. 6–7

3.3 Training and Testing Setup

  • Training proceeds in two stages: projector-only pretraining with the perception model and LLM frozen, followed by instruction tuning with LoRA on the LLM and segmentation losses.Pretraining adds projector regularization to text regression; instruction tuning supervises [SEG]-decoded masks with cross-entropy and dice losses. At testing, task inputs are tokenized and outputs decoded according to the task.Source: §3.3 · p. 7

4 Experiment

OMG-LLaVA combines image-, object-, and pixel-level understanding and reasoning in one model, with reported performance comparable to specialized methods across several tasks. Experiments highlight its referring-segmentation and grounded-conversation results, and show gains from perception prior embedding and foreground object queries.

4 Experiment

  • The instruction-tuning data span image-level LLaVA tasks, object-level Osprey and MDVP tasks, and pixel-level referring segmentation, semantic segmentation, and grounded conversation generation.The described datasets contain 665K image-level examples, approximately 74K Osprey and 200K MDVP examples, 74K referring-segmentation examples, 26K semantic-segmentation examples, and 200K GranDf examples.Source: §4 · p. 8

4.1 Main Results

  • OMG-LLaVA offers the most comprehensive capabilities among the compared MLLMs, with performance comparable to state of the art in referring segmentation, grounded conversation generation, and region captioning.It also retains basic image and video segmentation ability and uses a single visual encoder.Source: Table 2 · p. 8, §4.1 · p. 9
  • On validation sets, OMG-LLaVA exceeds LISA by 1.5, 3.2, and 4.3 cIoU on refCOCO, refCOCO+, and refCOCOg, respectively, with the OMG decoder frozen.With the decoder unfrozen and task-specific finetuning, it scores 78.0, 69.1, and 72.9 cIoU on those datasets, exceeding LISA by 3.1, 4.0, and 5.0 cIoU; it improves over PixelLM by 5.0 cIoU on refCOCO and 3.6 cIoU on refCOCOg.Source: Table 3 · p. 8, §4.1 · p. 9
  • With comparable training data, OMG-LLaVA surpasses LISA by 1.9 METEOR and 7.3 CIDEr in image description, and by 4.7 AP50 and 3.5 mIoU in pixel understanding.Despite GLaMM using much more training data, OMG-LLaVA outperforms it by 0.6 CIDEr, 1.4 AP50, and 0.1 mIoU on the test set.Source: §4.1 · p. 9

4.2 Ablation and Analysis

  • Perception prior embedding improves RES by 13.8, 10.6, and 11.7 cIoU on refCOCO, refCOCO+, and refCOCOg, and improves GCG by 11.1 mIoU.Adding foreground object queries further improves performance by 1.9 cIoU on refCOCO and 1.5 mIoU on GCG; the baseline struggled because the LLM lacked segmentation priors.Source: Table 5 · p. 8, §4.2 · pp. 9–10
  • Visualizations show perception prior embedding strengthens text–segmentation association, while object query input slightly improves some challenging segmentation cases.The study illustrates the strategies with baseline, perception-prior, and combined models.Source: Figure 6 · p. 9, §4.2 · p. 10
  • Qualitative visualizations cover OMG-LLaVA outputs on image-, object-, and pixel-level tasks, with additional referring-segmentation and grounded-conversation examples in the appendix.Source: §4.2 · p. 10

5 Conclusion

OMG-LLaVA combines image-, object-, and pixel-level understanding and reasoning in one model built from a single image encoder, LLM, and decoder. With perception prior embedding and unified task instruction tuning, it supports over eight multimodal tasks while preserving OMG-Seg’s visual perception ability.

5 Conclusion

  • OMG-LLaVA achieves comparable results to previous combined works with fewer trainable parameters and lower computation costs.The authors hope this work will inspire MLLM designs that minimize model components while maximizing functionality.Source: §5 · p. 10
  • With perception prior embedding and unified task instruction tuning, OMG-LLaVA performs over eight multimodal learning tasks while preserving the visual perception ability of the OMG-Seg baseline.Source: §5 · p. 10
  • OMG-LLaVA bridges image-level, object-level, and pixel-level understanding and reasoning using one image encoder, one LLM, and one decoder.Source: §5 · p. 10

A Appendix

The appendix provides implementation and training details, expanded ablations and visualizations, and discussion of limitations and future work.

A Appendix

A.1 More Implementation Details

OMG-LLaVA is trained in a projector-focused pre-training stage followed by instruction tuning, then uses task-specific inference prompts and output handling.

Pre-training

  • Pre-training freezes the visual encoder, OMG head, and LLM while training the visual and text projectors for one epoch on the LLaVA pre-training data.The batch size is 256 (32 per GPU), and the learning rate is 0.001.Source: §A.1 · p. 16

Supervised fine-tuning

  • Instruction tuning freezes the visual encoder and OMG head, applies LoRA to the LLM, and fully fine-tunes both projectors for one epoch across six dataset types.The datasets cover LLaVA instruction tuning, referring expression and semantic segmentation, grounded conversation generation, and mask-based and point-based visual prompts; batch size is 128 (16 per GPU), with a 2e-4 learning rate.Source: §A.1 · p. 16

Inference details for each task

  • At inference, OMG-LLaVA generates answers token by token, uses fixed questions for referring-expression segmentation and grounded conversation, and removes special tokens for other tasks.If the [SEG] token is absent in rare referring-segmentation cases, the system returns an empty mask.Source: §A.1 · p. 16

A.2 More Experiment Results.

OMG-LLaVA reports strong results across image-level benchmarks under joint and LLaVA-only training, and the Qwen2-7B variant improves on InternLM2 across pixel- and image-level benchmarks.

Evaluation results on image-level benchmarks

  • 1412 MME, 47.9 MMBench, 46.5 SEED-Bench, 80.0 POPE, and 42.9 AI2D are reported for jointly co-trained OMG-LLaVA, which improves over GLaMM, PixelLM, and LISA.These results use joint co-training on image-level and pixel-level datasets.Source: §A.2 · p. 16
  • 1730 MME, 67.5 MMBench, 68.9 SEED-Bench, 89.7 POPE, and 61.7 AI2D are achieved with LLaVA-only training.With the same training data, OMG-LLaVA exceeds LLaVA-1.5 by 41 on MME, 3.0 on SEED-Bench, 3.0 on POPE, and 5.1 on AI2D.Source: §A.2 · p. 17

Performance with diverse LLMs

  • 76.7 and 69.1 cIoU on RefCOCO and RefCOCO+ are achieved with Qwen2-7B, which outperforms InternLM2 on pixel- and image-level benchmarks.Qwen2-7B also scores 1466.8 MME, 62.8 MMBench, 60.7 SEED-Bench, 84.3 POPE, 52.6 AI2D, 37.2 MMstar, and 66.4 SQA.Source: §A.2 · p. 17

A.3 More Detailed Ablation Studies.

The ablations favor a shared MLP projector for object-centric tokens and a unified flexible answer format for segmentation tasks. Using the [SEG] token’s last-layer hidden state is sufficient for segmentation embeddings.

Projector for object-centric visual tokens

  • Adding cross-attention or unsharing the projector reduced segmentation and visual prompt-based task performance, supporting a shared MLP projector for object-centric visual tokens.Cross-attention introduced too many pixel-centric tokens, interfering with object information.Source: §A.3 · p. 17

Answer format for segmentation-based tasks

  • Unifying segmentation answers as “<p> Expression </p> [SEG]” improved referring expression segmentation over LISA’s fixed answer [47] while avoiding harm to instruction following.GLaMM’s flexible expression format [85] motivated testing a variable response rather than a fixed reply.Source: §A.3 · pp. 17–18

Segmentation embeddings

  • Using all-layer mean hidden states for [SEG] yielded negligible improvement on refCOCO but a significant drop on refCOCOg, while concatenating layers significantly reduced performance across RES tasks.The last-layer [SEG] hidden state already contains sufficient features for generating the segmentation mask.Source: §A.3 · p. 18

A.4 More Visualization Results

The visualizations show OMG-LLaVA combining segmentation with scene understanding, instruction following, and flexible visual-prompt interaction. Across tasks, the paper reports qualitative advantages over LISA and GLaMM and additional capabilities on RES and GCG examples.

Qualitative comparison with SOTA methods

  • On RES, OMG-LLaVA shows more stable segmentation than LISA and GLaMM, alongside better image and text understanding than LISA (13B) and GLaMM.The passage illustrates the understanding comparison with the phrase “the smallest chair.”Source: §A.4 · p. 18
  • On GCG visualizations, OMG-LLaVA gives more detailed, accurate scene descriptions than GLaMM and more stable segmentation, including for a mountain example.The descriptions include objects such as a lighthouse and a bear.Source: §A.4 · p. 19
  • For visual-prompt-based description, OMG-LLaVA accepts point, box, and mask prompts and produces more detailed captions and more accurate image understanding than GLaMM.Source: §A.4 · p. 19

Image-based conversation

  • On image-based conversation, OMG-LLaVA follows instructions more strongly than LISA and GLaMM, while giving more detailed answers than GLaMM and showing stronger scene reasoning than LISA.Both comparison models incorrectly segmented the athlete’s jersey in one example; LISA gave an incorrect answer to another question despite using a 13B LLM.Source: §A.4 · p. 19

Visualization results of RES

  • Additional RES visualizations show accurate, reliable segmentation based on spatial relationships and human actions, as well as reasoning segmentation without training on reasoning segmentation data.In the reasoning example, the model infers the target from a question and segments the corresponding object.Source: §A.4 · p. 19

Visualization results of GCG

  • GCG visualizations show strong scene understanding and object segmentation, though some objects are missed when Grandf captions omit them.The authors suggest higher-quality training data could improve performance.Source: §A.4 · p. 19

Visualization results of visual prompts-based description

  • Further visual-prompt examples show that point, box, and mask inputs elicit detailed descriptions of objects and their relationships in the scene.Source: §A.4 · p. 19

A.5 Limitation and Future Work Discussion

OMG-LLaVA has limitations in preserving image-level capability during joint pixel-level training and in supporting part-level segmentation. Future work includes video reasoning and broader instruction tuning, with data availability and computation as constraints.

Limitations

  • Joint training with pixel-level data can decrease image-level capability, as also observed in LISA [47] and GLaMM [85].The paper suggests organizing the data to eliminate this conflict.Source: §A.5 · p. 19
  • OMG-LLaVA cannot perform part-level segmentation because OMG-Seg lacks multi-granularity segmentation capability.A more powerful universal perception module with part-level visual inputs could address this limitation.Source: §A.5 · p. 19

Future Works

  • Although OMG-Seg can accept video inputs, OMG-LLaVA cannot perform pixel-level spatial-temporal reasoning because such datasets are lacking.The paper identifies video as a future direction.Source: §A.5 · p. 19
  • More instruction-tuning data with localization outputs and multi-round conversations could strengthen the MLLM; planned co-training with full GLaMM [85] and detection datasets depends on more computation resources.The paper proposes these datasets as future work if additional compute is available.Source: §A.5 · p. 19

OMG-LLaVA · GLaMM

The examples show OMG-LLaVA describing a baseball scene, while GLaMM passages provide shorter and LISA (13B) descriptions and identify qualitative visualizations for several tasks.

OMG-LLaVA

  • OMG-LLaVA describes a baseball game with a batter, catcher, other players and spectators, and a bench in the background.The passage also notes spectators at different distances from the field.Source: OMG-LLaVA · p. 22

GLaMM

  • GLaMM gives a concise description of a baseball player holding a bat on a field.Source: GLaMM · p. 22
  • LISA (13B) describes a batter preparing to swing and reports six people and a baseball glove in the scene.It also characterizes teammates and opponents as focused on the game.Source: GLaMM · p. 22
  • Figure 10 presents a qualitative comparison on the image-based conversation task.Source: Figure 10 · p. 22
  • Figure 11 shows additional visualization results for referring expression segmentation.Source: Figure 11 · p. 23
  • Figure 12 presents visualization results for reasoning segmentation.Source: Figure 12 · p. 23
  • Figures 13 and 14 show additional visualization results for grounded conversation generation.Source: Figure 13 · p. 24, Figure 14 · p. 24

Related papers

Loading 2406.19389v2…