PaperlayerView exact revision on arXiv

Source-linked AI summary

OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding

Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji, Chen Change Loy, Shuicheng Yan

arXiv:2406.19389v2cs.CV

TL;DR

Existing multimodal models provide reasoning but limited pixel-level understanding, while segmentation systems lack comparable reasoning and text-controlled interaction. OMG-LLaVA combines a universal perception module with an LLM, using perception prior embedding and unified instruction tuning to process image, text, and visual prompts. It unifies image-, object-, and pixel-level capabilities, with competitive results across segmentation and grounded conversation benchmarks, while retaining documented limitations in image-level trade-offs, part-level segmentation, and video spatial-temporal reasoning.

  • Problem

    Existing multimodal models lack precise pixel-level understanding and visual-prompt flexibility, while universal segmentation methods lack reasoning and text instruction control.

  • Method

    OMG-LLaVA combines a frozen universal perception module with one LLM, using perception prior embedding and unified instruction formation for image, text, and visual-prompt inputs.

  • Results

    OMG-LLaVA bridges image-, object-, and pixel-level understanding and reasoning, supports over 8 multimodal tasks, and achieves competitive results across multiple benchmarks.

  • Takeaways & Limitations

    A single encoder, decoder, and LLM can provide broad multimodal functionality while using fewer trainable parameters and lower computation costs than combined works.

  • Takeaways & Limitations

    Joint pixel-level training can reduce image-level capability, and the model lacks part-level segmentation and pixel-level spatial-temporal video reasoning.

Abstract

from arXiv · show

Current universal segmentation methods demonstrate strong capabilities in pixel-level image and video understanding. However, they lack reasoning abilities and cannot be controlled via text instructions. In contrast, large vision-language multimodal models exhibit powerful vision-based conversation and reasoning capabilities but lack pixel-level understanding and have difficulty accepting visual prompts for flexible user interaction. This paper proposes OMG-LLaVA, a new and elegant framework combining powerful pixel-level vision understanding with reasoning abilities. It can accept various visual and text prompts for flexible user interaction. Specifically, we use a universal segmentation method as the visual encoder, integrating image information, perception priors, and visual prompts into visual tokens provided to the LLM. The LLM is responsible for understanding the user's text instructions and providing text responses and pixel-level segmentation results based on the visual information. We propose perception prior embedding to better integrate perception priors with image features. OMG-LLaVA achieves image-level, object-level, and pixel-level reasoning and understanding in a single model, matching or surpassing the performance of specialized methods on multiple benchmarks. Rather than using LLM to connect each specialist, our work aims at end-to-end training on one encoder, one decoder, and one LLM. The code and model have been released for further research.

OMG-LLaVA

The paper is authored by Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, and Shengqiong Wu, among others.

  • Tao Zhang is listed among the paper’s authors.
  • Xiangtai Li is listed among the paper’s authors and is marked as project leader.
  • Shunping Ji, Chen Change Loy, and Shuicheng Yan are also listed as authors.

1 Introduction

OMG-LLaVA addresses the gap between multimodal models’ reasoning and universal segmentation models’ pixel-level capabilities by unifying image-, object-, and pixel-level understanding in one architecture.

  • OMG-LLaVA: OMG-LLaVA is presented as handling diverse image-level, object-level, and pixel-level understanding and reasoning tasks.
  • Motivation and Our Goals: LLaVA-like models support image-level visual reasoning but cannot perform fine-grained object-level and pixel-level understanding.
  • Motivation and Our Goals: The paper asks whether one MLLM can bridge image-, object-, and pixel-level tasks using one LLM, visual encoder, and visual decoder.
  • OMG-LLaVA: OMG-LLaVA preserves OMG-Seg’s pixel-level segmentation while adding referring segmentation, reasoning segmentation, and grounded conversation through text processing.
  • OMG-LLaVA: The model supports image-level captioning and conversation, visual-prompt-based object understanding, and region-level captions with one LLM, encoder, and decoder.
  • OMG-LLaVA: Perception prior embedding and unified instruction formation support image, text, and visual-prompt inputs with text, segmentation-token, mask, and label outputs.

2 Related Work

Related work spans multimodal language models, unified segmentation, language-driven localization and segmentation, and visual prompting, motivating a simpler unified alternative.

  • Multimodal Large Language Models: Multimodal language models increasingly use visual features as tokens and instruction tuning to address multiple benchmarks.
  • Unified Segmentation Models: Unified segmentation research develops end-to-end mask-classification models for image, video, and generalization segmentation tasks.
  • Language-driven Location and Segmentation: Language-driven segmentation work includes referring segmentation, localization, reasoning, region-level captioning, and systems that assign visual experts.
  • Capability Comparison: The capability comparison presents OMG-LLaVA as covering image-, object-, and pixel-level understanding and reasoning with a single visual encoder.
  • Language-driven Location and Segmentation: OMG-LLaVA is positioned as a more elegant baseline than approaches using multiple visual experts, with one visual encoder, one LLM, and one decoder.
  • Visual Prompts: Visual prompting methods explore learnable tokens, mask visual modeling, and prompting encoders for visual outputs; OMG-LLaVA supports visual prompts for user interaction.

3 Methodology

OMG-LLaVA unifies image-, object-, and pixel-level understanding and reasoning as token-to-token generation using one visual perception module and one LLM. A frozen OMG-Seg module supplies pixel- and object-centric tokens, while the LLM produces text and segmentation outputs from text, image, and visual prompts.

  • Motivation: LLaVA-like models support visual reasoning but lack fine-grained object-level and pixel-level understanding, motivating a unified architecture.
  • Unified task representation: The framework unifies image captioning, image conversation, region captioning, visual prompt conversation, referring segmentation, reasoning segmentation, and grounded conversation generation.These tasks are modeled as token-to-token generation with one visual perception module and one large language model.
  • Unified task representation: OMG-LLaVA represents text, dense image information, and specified-object information with text, pixel-centric, and object-centric visual tokens.Object-centric tokens can be decoded into segmentation masks.
  • OMG-LLaVA framework: A frozen universal perception module encodes images and visual prompts into pixel- and object-centric tokens, while the LLM outputs text responses and object-centric tokens for mask decoding.The OMG decoder remains part of the frozen perception module and converts object information into explicit segmentation responses.
  • OMG-LLaVA framework: The OMG decoder uses learnable object queries and visual-prompt queries with masked cross-attention and self-attention to produce object-centric tokens decodable into masks and categories.Attention-mask constraints preserve box and mask prompt information instead of relying only on point conversion.
  • Perception prior embedding: Perception prior embedding derives mask scores from segmentation masks and confidence scores, adds weighted object queries to image features, and forms pixel-centric visual tokens.This integrates object-level perception priors into image features without fine-tuning the perception module.

4 Experiment

OMG-LLaVA is evaluated across multimodal understanding, segmentation, and grounded conversation tasks, where it shows broad capabilities and competitive performance. Ablations further attribute gains to perception prior embedding and object query inputs.

  • Main Results: OMG-LLaVA achieves comprehensive pixel-level and object-level capabilities, including universal segmentation, referring segmentation, grounded conversation generation, and region captioning.It uses a single visual encoder in its system design.
  • Referring Expression Segmentation: 1.5, 3.2, and 4.3 cIoU improvements over LISA are reported on refCOCO, refCOCO+, and refCOCOg validation sets, respectively.These results use a frozen OMG decoder and a single visual encoder.
  • Grounded Conversation Generation: OMG-LLaVA surpasses LISA by 1.9 METEOR, 7.3 CIDEr, 4.7 AP50, and 3.5 mIoU on grounded conversation generation.It also exceeds GLaMM by 0.6 CIDEr, 1.4 AP50, and 0.1 mIoU on the test set despite GLaMM using more training data.
  • Ablation Study: Adding foreground object queries yields further gains of 1.9 cIoU on refCOCO and 1.5 mIoU on GCG.Visualization attributes the improvement to better handling of some challenging segmentation cases.
  • Evaluation Setup: The experiments include image-level, object-level, and pixel-level qualitative visualizations alongside multiple segmentation and grounded conversation evaluations.The study uses LLaVA, object-level, and pixel-level datasets spanning diverse multimodal tasks.

5 Conclusion

The paper presents OMG-LLaVA as a unified model for image-, object-, and pixel-level understanding and reasoning. Its design uses one encoder, one LLM, and one decoder while supporting more than eight multimodal tasks.

  • Conclusion: OMG-LLaVA bridges image-level, object-level, and pixel-level understanding and reasoning in one model.The framework uses one image encoder, one LLM, and one decoder.
  • Conclusion: The model performs over 8 multimodal learning tasks while preserving the visual perception ability of the OMG-Seg baseline.The authors report comparable results to previous combined works with fewer trainable parameters and lower computation costs.

A Appendix

The appendix provides additional implementation and training details, component ablations, visualization results, and discussion of limitations and future work.

  • Appendix: The appendix covers implementation and training details, detailed component ablations, visualization results, and limitations and future work.

A.1 More Implementation Details

OMG-LLaVA uses a staged training procedure that first aligns visual tokens with text space and then instruction-tunes the language model and projectors across diverse multimodal datasets.

  • Pre-training: During pre-training, the visual and text projectors are trained for visual-text alignment while the visual encoder, OMG head, and LLM remain frozen.The stage uses the LLaVA pretraining data for 1 epoch.
  • Supervised Fine-tuning: During instruction tuning, the visual encoder and OMG head remain frozen, the LLM is fine-tuned with LoRA, and the text and visual projectors are fully fine-tuned.
  • Supervised Fine-tuning: Instruction tuning uses image-level, referring segmentation, semantic segmentation, grounded conversation, mask-based visual prompt, and point-based visual prompt datasets.
  • Inference: At inference, OMG-LLaVA generates answers token by token from the question and uses fixed templates for referring segmentation and grounded conversation generation.If no [SEG] token is predicted for referring segmentation, an empty mask is returned.

A.2 More Experiment Results.

OMG-LLaVA performs strongly across image-level benchmarks, with results varying by training data and language model choice. Additional ablations examine projector and answer-format design.

  • Image-level benchmarks: 1412, 47.9, 46.5, 80.0, and 42.9 on MME, MMBench, SEED-Bench, POPE, and AI2D show strong image-level performance with joint co-training.The paper reports significant improvement over GLaMM, PixelLM, and LISA.
  • Ablations: The answer-format ablation compares fixed and unified formats for segmentation-based tasks.The supplied table caption identifies the fixed formats and the unified “<p> Expression </p> [SEG]” format.
  • Image-level benchmarks: 1730, 67.5, 68.9, 89.7, and 61.7 on MME, MMBench, SEED-Bench, POPE, and AI2D result from training only on the LLaVA dataset.With the same training data, OMG-LLaVA exceeds LLaVA-1.5 by 41 on MME, 3.0 on SEED-Bench, 3.0 on POPE, and 5.1 on AI2D.
  • Language-model comparison: Qwen2-7B produces better pixel-level and image-level benchmark performance than InternLM2 in the reported language-model comparison.The supplied passage also reports 76.7 cIoU and 69.1 cIoU on RefCOCO and RefCOCO+ with Qwen2-7B.

A.3 More Detailed Ablation Studies.

The ablations show that preserving distinct object-centric tokens, using a flexible unified answer format, and relying on final-layer segmentation embeddings support the model’s segmentation behavior.

  • Projector design: Adding cross-attention to the object-centric visual-token projector decreases segmentation and visual prompt-based task performance.The paper attributes this to object-centric tokens incorporating too many pixel-centric visual tokens.
  • Answer format: The unified “<p> Expression </p> [SEG]” answer format improves referring expression segmentation and avoids damage to instruction-following ability.The fixed answer format can cause responses to overfit to “Sure, it is [SEG].”
  • Segmentation embeddings: Using the mean of all-layer [SEG] hidden states gives negligible refCOCO improvement but significantly degrades refCOCOg performance.Concatenating all-layer hidden states causes significant performance drops across RES tasks.
  • Segmentation embeddings: The last-layer [SEG] hidden state already contains sufficient features to generate the segmentation mask.Adding intermediate-layer hidden states does not yield better segmentation results.

A.4 More Visualization Results

Qualitative results portray OMG-LLaVA as stable across segmentation, grounded conversation, visual-prompt description, and image-based conversation tasks. The visualizations emphasize detailed descriptions, flexible prompts, and instruction-following.

  • Referring expression segmentation: OMG-LLaVA shows more stable referring-expression segmentation than LISA and GLaMM in the qualitative comparison.The comparison also reports better image and object understanding.
  • Grounded conversation generation: OMG-LLaVA provides more detailed scene descriptions and more stable segmentation than GLaMM in grounded conversation generation.Examples include identifying a “lighthouse” and “bear,” and more stable segmentation of a “mountain.”
  • Visual prompt-based description: OMG-LLaVA accepts point, box, and mask visual prompts and produces more detailed object captions than GLaMM.The visualization reports more accurate image understanding alongside this prompt flexibility.
  • Image-based conversation: OMG-LLaVA demonstrates stronger instruction-following, scene understanding, and reasoning than LISA and GLaMM in image-based conversation examples.For one question, it correctly answers that the jersey number is 52 while the comparison systems incorrectly segment the jersey.
  • Additional visualizations: Additional visualizations show reasoning segmentation without training on reasoning-segmentation data.The reported examples rely on understanding spatial relationships and human actions.
  • Additional visualizations: Grounded conversation visualizations show strong scene understanding and object segmentation, while omitted caption objects can be overlooked.The paper attributes some omissions to incomplete image captions in the Grandf dataset.
  • Additional visualizations: Further visual-prompt examples combine point, box, and mask inputs with descriptions of objects and their relationships.These examples reinforce the model’s support for multiple visual-prompt types.

A.5 Limitation and Future Work Discussion

The paper identifies limitations in joint image-level and pixel-level training, part-level segmentation, and pixel-level spatial-temporal reasoning. Additional examples and figures illustrate the model’s conversation and visual-prompt outputs.

  • Limitations: Joint training with pixel-level understanding data can decrease image-level capability.The paper notes this phenomenon is also observed in LISA and GLaMM.
  • Limitations: OMG-LLaVA cannot perform part-level segmentation because OMG-Seg lacks multi-granularity segmentation capability.The proposed direction is a more powerful universal perception module with part-level visual inputs.
  • Future work: OMG-LLaVA cannot yet perform pixel-level spatial-temporal reasoning on video because suitable datasets are lacking.Future work includes video and additional instruction-tuning data.
  • Additional examples: The reported image examples include detailed baseball descriptions, jersey-number answering, and catcher-role reasoning.One example identifies the jersey number as 52, while another explains that the squatting athlete is preparing to catch the ball.
  • Additional figures: Figures 10–14 provide qualitative examples for image conversation, referring segmentation, reasoning segmentation, and grounded conversation generation.The supplied figure captions identify the tasks but do not state additional quantitative outcomes.
Loading 2406.19389v2…