Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic

Referential Dialogue

• RD lets users point to image regions and ask about them, while the model can indicate precise locations in its replies when needed.The introduction presents this interaction as a familiar part of everyday human communication.Source: §1 · pp. 1–2, Figure 1 · p. 1

Applications

• The paper identifies applications in XR assistants that display referenced areas, visual robots that understand reference positions, and online shopping inquiries about image items.These are presented as potential uses of referential dialogue.Source: §1 · pp. 1–2

Shikra

• Shikra handles spatial coordinates as natural-language numbers, using a vision encoder, alignment layer, and LLM without extra vocabularies, position encoders, detection modules, or external plug-ins.The paper describes this as a simple, unified architecture.Source: §1 · p. 2

Research questions

• The paper also examines image-position representation, MLLMs’ understanding of absolute positions, and whether location-aware reasoning improves answer accuracy.These questions are presented as directions for analysis and potential inspiration for future MLLM research.Source: §1 · p. 2

Capabilities

• Shikra is reported to handle unseen settings and achieve promising performance on REC, PointQA, VQA, and image captioning without finetuning.The introduction describes RD as a superset of vision-language tasks and points to later quantitative results.Source: §1 · p. 2

2.1 Multimodal Large Language Model

• Multimodal LLMs use visual adapters and large-scale image-text training, instruction tuning, Q-Former alignment, or direct visual-feature projection; related work also extends LLMs to audio and control.Examples include Flamingo (Alayrac et al., 2022), MM-GPT (Gong et al., 2023), BLIP-2 (Li et al., 2023b), LLaVA (Liu et al., 2023a), and PaLM-E (Driess et al., 2023).Source: §2.1 · p. 2

2.2 Vision-Language Positioning Tasks

• Positioning tasks cover locating described objects, grounding answers, describing input regions, generating discriminative referring expressions, and answering questions about points.REC (Mao et al., 2016) localizes a described target, while Described Object Detection handles absent or multiple objects; Shikra flexibly handles position inputs and outputs simultaneously, enabling Referential Dialogue.Source: §2.2 · pp. 2–3

2.3 Position Representation

• Position inputs use cropped patches, masks, or positional encodings, while outputs range from anchors and box regression to sequence generation with extra coordinate vocabularies.Unlike those coordinate-vocabulary approaches, Shikra represents position input and output directly in natural language; Pix2seq (Chen et al., 2021) uses 1,000 bins and tokens.Source: §2.3 · p. 4

3 Referential Dialogue

• In a user demonstration, Shikra compared two pointed-to deer, described their differences, and output coordinates for those differences.The paper reports that, to the authors’ knowledge, no unified model had achieved this functionality before.Source: §3 · p. 4
• Referential dialogue covers conventional tasks such as VQA, image captioning, and multimodal dialogue, as well as REC, REG, and PointQA.The paper characterizes RD as a superset of numerous vision-language tasks.Source: §3 · p. 4
• Shikra demonstrates abilities absent from its training set, including comparing similarities between indicated objects, counting, and providing object positions.Source: §3 · p. 4

4 Chessboard Test for Current MLLM

• The test divides each image into a 2 × 2 chessboard and asks a model to choose which quadrant contains the named object.The test simplifies object grounding into a four-way part-choice task because current MLLMs cannot directly output coordinates.Source: §4 · p. 4
• Using LVIS (Gupta et al., 2019), the authors selected 600 images per part, totaling 2,400 images across 945 categories; objects completely within a part were chosen, excluding ambiguous positions.Source: §4 · p. 4
• LLaVA-13B achieved 25.96% accuracy, comparable to random selection, suggesting coarse-grained vision-language alignment may not capture exact image positions.The authors argue that appropriate coordinate representations and finer-grained training data need to be explored.Source: §4 · p. 4

5.1 Architecture

• Shikra uses CLIP ViT-L/14, a fully connected alignment layer, and Vicuna-7B or Vicuna-13B, without position-specific vocabulary, encoders, or detection modules.The layer maps 16 × 16 × 1024 visual embeddings to 256 × D, where D is 4,096 for Vicuna-7B and 5,120 for Vicuna-13B; it and the full LLM are trained.Source: §5.1 · pp. 4–5

5.2 Numerical representation of position

• Positions are ordinary natural-language numbers: normalized [xmin, ymin, xmax, ymax] boxes or [xcenter, ycenter] points, typically to three decimal places.Coordinates can appear anywhere in input and output sequences and are tokenized like regular text.Source: §5.2 · p. 5

5.3.1 Reorganization of public data

• Reorganized public data covers VQA, image captioning, REC/REG, grounding captions, PointQA, and Spotting Captioning, drawing on RefCOCO (Kazemazdeh et al., 2014), Visual Genome (Krishna et al., 2017), and Visual-7W (Mani et al., 2020).Test and validation images were excluded from training to prevent data leakage.Source: §5.3.1 · p. 5

5.3.2 Generated data

• Because existing public data lacked positional-CoT and natural-communication annotations, GPT-4 used Flickr30K Entities’ five descriptions and labeled boxes to generate Shikra-RD’s 5,922 Q&A pairs, with coordinates possibly in both questions and answers.Source: §5.3.2 · p. 5

5.3.3 Task prompts

• Task prompts use varied natural-language templates, which GPT-4 rewrites into hundreds of variations so training can sample them randomly.The paper notes that a fixed template would limit the model’s ability to flexibly accept user instructions.Source: §5.3.3 · pp. 5–6

5.4 Tuning details

• Training has two stages: 100,000 steps on reorganized VL data, then 50% sampling of LLaVA-Instruct-150K (Liu et al., 2023a) and generated RD data.The visual encoder is frozen and all LLM parameters are tuned in both stages; training takes around 100 hours and 20 hours, respectively.Source: §5.4 · p. 6

6.1 Grounding CoT or verbal CoT?

• 13 points over Q→CA and 5.9 points over Q→A were achieved with center-point-annotated Grounding CoT, while verbal CoT alone reduced performance versus direct answering.The authors interpret the improvement as evidence that positional annotations suppress visual hallucination, describing this as a preliminary attempt at Grounding CoT.Source: §6.1 · p. 6

6.2 Location tokens or just numbers?

• Shikra’s direct numerical coordinate representation achieves better REC results than extra coordinate vocabularies and permits users to adjust precision without retraining vocabularies.The comparison uses two toy Shikra models trained with different representations; the authors also describe the numerical representation as simpler.Source: §6.2 · p. 6, Table 2 · p. 6
• Numerical coordinates require more tokens than extra vocabularies, increasing computational costs when predicting dense objects.The authors nevertheless prefer numerical representation while noting that future work may choose between the methods based on their pros and cons.Source: §6.2 · p. 7

6.3 Quantitative results on conventional tasks

• On REC, Shikra shows promising performance compared with other generalist models when instructed to return coordinates for the described object.The evaluation also compares against localization specialists and foundation models finetuned on localization-related tasks.Source: §6.3 · p. 7, Table 3 · p. 7
• Shikra achieves state-of-the-art performance across all evaluated LookTwice-QA and Visual7W PointQA settings.LookTwice-QA questions concern a user-designated point or box at different levels of referential clarity; Visual7W PointQA asks the model to choose among four boxes.Source: Table 5 · p. 7, Table 4 · p. 7, §6.3 · p. 8
• Shikra reports promising results on most evaluated VQA and image-captioning datasets, and POPE results comparable to InstructBLIP and far above recent popular MLLMs.VQA and image captioning require no coordinates in input or output; the POPE comparison is reported using its evaluation pipeline.Source: §6.3 · p. 8, Table 6 · p. 8, Table 7 · p. 8

• Shikra supports only English, making it unfriendly to non-English speakers; multilingual support is identified as valuable future work.Source: §7 · p. 9
• Shikra is unsuitable for dense object detection and segmentation, and improved coordinate representations for these tasks are a future direction.Source: §7 · p. 9
• Like most LLMs, Shikra may produce harmful and counterfactual responses.Source: §7 · p. 9

8 Conclusion

• Shikra addresses MLLMs’ gap in understanding and engaging in referential dialogue by comprehending and outputting spatial coordinates in natural language.The paper presents it as a unified, straightforward model.Source: §8 · p. 9
• Shikra requires no extra vocabularies, position encoders, or external plug-ins, preserving the model’s simplicity.Source: §8 · p. 9
• Shikra performs notably well on conventional vision-language tasks and enables applications such as Mixed Reality AI assistance and precise communication in online shopping.Source: §8 · p. 9

A Details of All Training Data

• The training data are listed in Table 8.The supplied passage does not specify the datasets or their quantities.Source: A Details of All Training Data · p. 11
• An asterisk marks data used only in the second training stage.Source: A Details of All Training Data · p. 11
• Images shared with the testing or validation set were removed from the training set to prevent potential data leakage.Source: A Details of All Training Data · p. 11

B Examples of Task Prompts

• Each listed training task has hundreds of prompts generated by GPT-4 using carefully designed instructions.The paper says it randomly selected three prompts to help readers understand them.Source: B Examples of Task Prompts · pp. 11–12
• Table 9 templates use <image>, <objs>, <question>, and <expr> for images, specified locations, VQA questions, and REC expressions, but inference is not restricted to these forms.Users can express their needs in natural language to create more diverse task formats.Source: Table 9 · p. 12
• Examples cover OCR, REC, REG, PointQA, and captioning, with Figures 9 and 10 showing that Shikra can handle points and boxes when instructed.The passage also references Figure 5 for OCR, Figure 8 for REC, Figure 7 for REG, and Figure 9 for PointQA.Source: B Examples of Task Prompts · p. 12

Referential Dialogue

• In a puzzle exchange, Shikra tells the user to place the identified missing piece into the empty space.Figure 3 distinguishes user- or jointly referred regions with dashed boxes from regions referred to only by Shikra with solid boxes.Source: Referential Dialogue · p. 13, Figure 3 · p. 13
• In Q→CBoxA, Shikra generates a grounded explanation for its answer.Source: Figure 4 · p. 13
• Shikra demonstrates OCR despite having no explicit OCR datasets in its training.Source: Figure 5 · p. 13
• In spotting captioning, Shikra describes an image and marks mentioned objects or regions with points or boxes.Source: Figure 6 · p. 14
• For referring expression generation, Shikra generates a unique description for a specified location.Source: Figure 7 · p. 14
• For referring expression comprehension, Shikra localizes the target object described by a referring expression.Source: Figure 8 · p. 15
• In PointQA, Shikra answers questions about user-specified regions, while PointQA-V7W asks it to choose among four box options.The region may be specified by a center point or box in PointQA.Source: Figure 9 · p. 15, Figure 10 · p. 15