Source-linked AI summary
Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, Rui Zhao
TL;DR
MLLMs lack the ability to conduct referential dialogue in which users point to image regions and models respond with spatially grounded references. Shikra addresses this gap with a unified natural-language coordinate interface and performs promisingly across referential and conventional VL tasks, while remaining limited in language coverage and dense vision tasks.
Problem
Current MLLMs cannot let users indicate image regions or provide exact locations for described content during dialogue.
Method
Shikra is a unified MLLM using a vision encoder, alignment layer, and LLM to process spatial coordinates as natural-language numbers without extra vocabularies, position encoders, detection modules, or plug-ins.
Results
Shikra achieves promising performance across Referential Dialogue and conventional VL tasks including REC, PointQA, VQA, and Image Captioning.
Takeaways & Limitations
Referential Dialogue provides a unified interaction capability that extends across location-related and conventional VL tasks and supports applications such as XR assistance and image-based shopping.
Takeaways & Limitations
Shikra supports only English, is unsuitable for dense object detection and segmentation, and may produce harmful or counterfactual responses like most LLMs.
Abstract
from arXiv · showhide
In human conversations, individuals can indicate relevant regions within a scene while addressing others. In turn, the other person can then respond by referring to specific regions if necessary. This natural referential ability in dialogue remains absent in current Multimodal Large Language Models (MLLMs). To fill this gap, this paper proposes an MLLM called Shikra, which can handle spatial coordinate inputs and outputs in natural language. Its architecture consists of a vision encoder, an alignment layer, and a LLM. It is designed to be straightforward and simple, without the need for extra vocabularies, position encoder, pre-/post-detection modules, or external plug-in models. All inputs and outputs are in natural language form. Referential dialogue is a superset of various vision-language (VL) tasks. Shikra can naturally handle location-related tasks like REC and PointQA, as well as conventional VL tasks such as Image Captioning and VQA. Experimental results showcase Shikra's promising performance. Furthermore, it enables numerous exciting applications, like providing mentioned objects' coordinates in chains of thoughts and comparing user-pointed regions similarities. Our code, model and dataset are accessed at https://github.com/shikras/shikra.
1 Introduction
Existing MLLMs can perceive images but cannot participate in dialogue that lets users indicate regions and receive spatially grounded replies. Shikra introduces unified referential dialogue through natural-language coordinate inputs and outputs, supporting broader VL tasks.
- Motivation: MLLMs cannot let users indicate image regions or provide exact locations for described content, unlike ordinary human referential communication.Referential Dialogue (RD) captures this interaction pattern and motivates applications including XR assistance, visual-robot communication, and image-based shopping.
- Approach: Shikra represents spatial coordinates in natural-language numerical form for both inputs and outputs, enabling verbal and spatial explanations.The model can answer questions while indicating relevant regions when necessary.
- Scope: Referential Dialogue is presented as a superset of many VL tasks, including VQA, image captioning, REC, and PointQA.The paper also studies position representation, absolute-position understanding, and whether location-aware reasoning improves answers.
- Approach: Shikra is a generalist MLLM for Referential Dialogue without extra vocabularies, detection modules, or external plug-ins.Its architecture uses a vision encoder, alignment layer, and LLM.
- Results: Shikra achieves promising performance on REC, PointQA, VQA, and Image Captioning without finetuning.The introduction presents this result alongside its ability to handle unseen settings and diverse application scenarios.
2 Related Works
Prior vision-language work covers multimodal instruction following and localization-oriented tasks, but these tasks typically specialize in either region inputs or region outputs. Shikra is positioned as a model that handles both flexibly within referential dialogue.
- Multimodal LLMs: Prior MLLMs include Flamingo, OpenFlamingo, MM-GPT, and Otter, which extend language models with visual adaptation or instruction-tuning approaches.The cited related-work passage identifies these systems as representative multimodal language-model developments.
- Localization tasks: REC localizes a target object from a referring expression, while related tasks extend localization to absent or multiple objects.These tasks represent vision-language settings whose outputs include spatial regions.
- Positioning tasks: PointQA asks a model to answer a visual question about a queried position, making location an input to the task.Other positioning tasks similarly associate answers or descriptions with specific regions.
- Shikra: Shikra handles position representations flexibly and simultaneously as both inputs and outputs, extending conventional positional tasks into Referential Dialogue.This capability distinguishes the proposed model's task coverage from the listed specialized settings.
3 Referential Dialogue
Shikra demonstrates referential dialogue through user-pointed interactions, responding with descriptions and coordinates while extending beyond conventional multimodal language tasks.
- Shikra responds to user-pointed objects by describing differences and outputting coordinates for those differences.A Figure 1 demonstration shows a user pointing to two deer and asking about their differences.
- Referential dialogue enables Shikra to perform conventional tasks such as VQA, image captioning, and multimodal dialogue.
- Shikra also handles REC, REG, and PointQA, which current multimodal language models cannot handle in the same way.
- Shikra generalizes to tasks outside training, including comparing indicated objects, counting objects, and providing their positions.
4 Chessboard Test for Current MLLM
The chessboard test evaluates whether current MLLMs understand absolute spatial positions, finding that LLaVA performs near random selection and motivating coordinate-focused representations and training data.
- The chessboard test divides images into four equal regions and asks the model to classify an object’s quadrant.The evaluation uses top-left, top-right, bottom-left, and bottom-right answer choices.
- The test uses LVIS objects that are completely contained within one quadrant, covering 2,400 images across 945 categories.It selects 600 images per quadrant and excludes ambiguous positions.
- 25.96% accuracy on the chessboard test leaves LLaVA-13B comparable to random selection.The reported expected accuracy is 25.96%.
- The result suggests coarse-grained vision-language alignment pre-training is inadequate for capturing exact image positions.The authors consequently call for appropriate coordinate representations and finer-grained training data.
5 Breeding Shikra
Shikra combines a vision encoder, alignment layer, and language model with natural-language coordinates, and is trained using reorganized VL data plus generated referential-dialogue data and varied task prompts.
- Structure design: Shikra’s structure uses a visual encoder, an alignment layer, and a language model without additional position vocabularies, encoders, or detectors.The implementation maps visual embeddings into the LLM input space with one fully connected layer.
- Position representation: Positions are represented as normalized numerical coordinates, using [xmin, ymin, xmax, ymax] for boxes and [xcenter, ycenter] for center points.Coordinates use three decimal places and may appear anywhere in the input or output sequence.
- Training data: The task mixture includes VQA, image captioning, positional datasets, REC, grounding caption, PointQA, and Spotting Captioning.Spotting Captioning requires describing the image and locating mentioned objects or regions with points or boxes.
- Training data: Training data combines reorganized public VL datasets with 5,922 GPT-4-generated question-answer pairs from Flickr30K Entities.The generated Shikra-RD data includes coordinate information in questions and answers.
- Task prompts: GPT-4 expands task templates into hundreds of linguistic variations so the model can accept flexible user instructions.The templates cover tasks such as PointQA and REC.
- Training strategy: Shikra trains in two stages while freezing the visual encoder and tuning all LLM parameters.The first stage uses reorganized VL data; the second raises sampling of LLaVA-Instruct-150K and generated referential-dialogue data to 50%.
6 Experiment and Analysis
Shikra’s experiments examine grounding with positional reasoning, coordinate representations, and performance across location-related and conventional vision-language tasks. Results support numerical coordinates and show promising task performance, while also identifying higher token costs for dense outputs.
- Grounding CoT or verbal CoT?: Grounded CoT with center-point annotations improved performance by 13 points over verbal CoT and 5.9 points over direct answering on CLEVR.The authors describe this as indicating that positional annotations suppress visual hallucination, while calling it a preliminary attempt.
- Location tokens or just numbers?: Numerical coordinates achieved better REC results than extra coordinate vocabularies in controlled Shikra-7B comparisons.The numerical representation also allows users to control coordinate precision without modifying the vocabulary.
- Location tokens or just numbers?: Numerical coordinate representation requires more tokens than extra vocabularies, increasing computational costs when predicting dense objects.The paper retains numerical coordinates but notes that future work may choose between representations based on their trade-offs.
- Quantitative results on conventional tasks: Shikra achieved promising REC performance compared with generalist models and specialist localization systems.The comparison includes generalist models, localization specialists, and models with localization-related finetuning.
- Quantitative results on conventional tasks: Shikra achieved state-of-the-art performance across the evaluated Visual-7W and LookTwice-QA PointQA settings.The settings require answering questions associated with user-specified points or boxes, with varying levels of referential clarity.
- Quantitative results on conventional tasks: Shikra achieved promising results on most VQA and image-captioning datasets and was comparable to InstructBLIP while surpassing recent popular MLLMs on POPE.These conventional tasks do not require coordinates in their inputs or outputs.
7 Limitations
Shikra has several stated limitations involving language coverage, task suitability, and response safety.
- Shikra supports only English, limiting its user-friendliness for non-English speakers.
- Shikra is unsuitable for dense object detection and segmentation tasks.
- The model may produce harmful and counterfactual responses, like most LLMs.
8 Conclusion
The paper identifies a gap in MLLMs’ referential dialogue abilities and introduces Shikra to address it with natural-language spatial coordinates. Shikra performs well across conventional vision-language tasks while supporting practical applications.
- The paper introduces Shikra, a unified model that understands and outputs spatial coordinates in natural language.
- Shikra avoids extra vocabularies, position encoders, and external plug-ins, preserving a straightforward model design.
- Shikra performs notably well on a variety of conventional vision-language tasks.
- The model supports applications including Mixed Reality assistants and precise communication in online shopping.
A Details of All Training Data
The paper lists Shikra’s training data and explains that duplicate images were removed from training when they appeared in testing or validation sets.
- Table 8 lists all training data used by Shikra.
- Asterisk-marked data is used only in the second training stage.
- Images overlapping with testing or validation sets were removed from training to prevent potential data leakage.
B Examples of Task Prompts
Shikra uses numerous GPT-4-generated task prompts and accepts natural-language requests beyond fixed templates. Examples cover diverse spatial and conventional vision-language tasks.
- Examples of Task Prompts: Shikra uses hundreds of prompts for every listed task, generated by GPT-4 with carefully designed instructions.
- Examples of Task Prompts: Table 9 explains placeholders for images, user-specified locations, VQA questions, and referring expressions in task templates.
- Examples of Task Prompts: During inference, users can describe their needs in natural language without being confined to the listed prompt forms.
- Examples of Task Prompts: The examples include OCR, REC, REG, PointQA, and captioning tasks, with inputs and outputs supporting points and boxes.
C More Conversations with Shikra
This section presents additional Shikra-7B dialogue records spanning referential dialogue, grounded explanations, OCR, captioning, referring expressions, and point-based visual question answering. The examples show natural-language responses that associate described objects or regions with spatial coordinates.
- Additional dialogue records: Additional records showcase Shikra-7B results for referential dialogue and VQA with grounded explanations.The section introduces RD results and VQA (Q→CBoxA), whose task asks models to generate a grounded explanation for the answer.
- Referential dialogue: In a puzzle dialogue, Shikra identifies the puzzle, missing piece, and empty space with box coordinates.The response instructs placing the missing piece into the empty space and provides coordinates for all three regions.
- Spotting captioning: For image description, Shikra names objects and locations using the requested [x1,y1,x2,y2] format.The example localizes a boy, fishing line, railing, and pier while describing their relationships in the scene.
- Referring and captioning tasks: The showcased tasks include spotting captioning, REG, and REC, linking descriptions or locations to points, boxes, or localized objects.Spotting captioning identifies mentioned objects or regions; REG describes a specified location; REC localizes an object from a referring expression.
- Additional vision-language tasks: Shikra-7B is also illustrated on PointQA, PointQA-V7W, and OCR, including OCR despite having no explicit OCR datasets in training.PointQA concerns a user-specified region, while PointQA-V7W uses a question with four box options and requires selecting one answer.