NExT-Chat: An LMM for Chat, Detection and Segmentation

• Earlier LMMs focused on holistic image understanding, and pix2seq’s discrete coordinate outputs struggle to represent fine-grained formats such as segmentation masks.Source: §1 · pp. 1–2
• Pix2emb represents locations as embeddings decoded into different formats, using <trigger> to initiate localization and <loc> as a placeholder for object-location embeddings.The embedding-based formulation treats localization as regression, allowing L1, IoU, and GIoU losses.Source: §1 · p. 2
• 87.7 accuracy vs. Shikra’s 86.9 on POPE-Random, 68.9 average cIoU vs. LISA’s 67.9 on RES, and 79.6 CIDEr vs. Kosmos-2’s 62.3 on RefCOCOg region captioning.Source: §1 · p. 2
• NExT-Chat combines chat, region input, detection, and segmentation in one LMM built on pix2emb.The paper also presents examples and results to demonstrate the method’s effectiveness.Source: §1 · p. 2

2.1. LMM

• LMMs use visual features for captions and question answering; Flamingo [1] uses cross-attention, while BLIP-2 [17] and Kosmos [11] feed visual features into LLMs as soft prompts.Source: §2.1 · p. 2
• Instruction-tuning approaches include MiniGPT-4 [46], LLaVA, Otter [15] [16], and LLaVA-1.5 [20], while earlier LMMs [1] [21] lack region understanding.Source: §2.1 · p. 2

2.2. LMM for Region Reasoning

• Region methods include GPT4ROI [44] feature inputs, Pix2seq [5] coordinate tokens, and extensions to vision-language tasks [36] [38] [32] [27]; Kosmos-2 [27] uses 32×32 image bins.Source: §2.2 · p. 2
• Shikra [4] reuses the original vocabulary for box coordinates but requires roughly 26 tokens per box; NExT-Chat instead decodes boxes and masks from <trigger> hidden states.Source: §2.2 · p. 3, Figure 2 · p. 3

3.1. LMM Architecture

• The LLaVA-like architecture projects 24×24 CLIP ViT-L/14@336px image patches into the LLM’s word-embedding dimension as visual tokens.The decoder-only LLM is Vicuna-1.5.Source: §3.1 · p. 3

3.2. Pix2Emb Method

• For detection, the <trigger> hidden state is passed to a 2-layer MLP box decoder that regresses coordinates, supervised by L1 and GIoU losses.The predicted box uses [x0, y0, x1, y1] coordinates.Source: §3.2 · p. 3
• For segmentation, the <trigger> hidden state is projected as a SAM prompt, optionally concatenated with a detected-box prompt, while SAM also receives the original image.Mask training uses IoU, Dice, and Focal losses, with β set to 20.Source: §3.2 · p. 3
• Cycle loss binds box encoder and decoder training.Source: Figure 3 · p. 4

3.3. Training Process

• Training proceeds through pre-training, instruction tuning, and segmentation; the final stage freezes most parameters, trains the LMM–SAM projector and SAM decoder, and uses RefCOCO, RefCOCO+, and RefCOCOg segmentation splits.The final-stage training takes 3 hours on 8 A100 (80G) GPUs.Source: §3.3 · p. 4

4. Experiment

• Quantitative comparisons cover POPE image hallucination, RES segmentation, REC detection, and RefCOCOg region captioning.The reported table scopes identify POPE, RES, REC, and RefCOCOg as the evaluation settings.Source: §4 · p. 4, Table 1 · p. 5, Table 2 · p. 5

4.1. Applications across Different Scenarios

• NExT-Chat localizes queried objects, including each of four bears and the sky, and handles grounding questions about a TV remote and a boat.The boat example uses an object location supplied as input.Source: §4.1 · pp. 4–5
• For region captioning, NExT-Chat describes the specified region rather than the whole image, including small flags and background trees.The examples indicate that descriptions are tailored to the provided bounding boxes.Source: §4.1 · p. 5
• Grounded captions reference two or three major image objects and show a reduced tendency to mention nonexistent objects.The object references are organized into coherent sentences.Source: §4.1 · p. 5
• In detailed reasoning, NExT-Chat infers a man's occupation from contextual cues, localizes relevant regions, and explains possible duties for each hypothesis.The example uses the man's uniform and horse as contextual cues.Source: §4.1 · p. 6

5.1. Hallucination

• NExT-Chat ranks best on POPE random and popular splits and second on the adversarial split.The comparison is against state-of-the-art multimodal models.Source: §5.1 · p. 6

5.2. Referring Expression Segmentation

• NExT-Chat improves on LISA-7B in six segmentation dataset splits, including a 4.5-point gain on RefCOCO+ testA, with 127k masks versus over tenfold larger datasets.Against non-LMM methods, it is highest or second-highest across splits except RefCOCO+ val.Source: §5.2 · p. 7

5.3. Referring Expression Comprehension

• NExT-Chat beats VILLA, UNITER, and TranVG on every REC split, but scores slightly below Shikra-7B despite a similar detection-training data recipe.The authors hypothesize that loss balancing and the lack of regression pretraining contribute to the Shikra gap; they argue regression tasks remain important, especially for embodied AI.Source: §5.3 · p. 7

5.4. Region Caption

• NExT-Chat achieves the best CIDEr among the RefCOCOg region-caption baselines and beats the four-shot Kosmos-2 version.The evaluation uses CIDEr and METEOR.Source: §5.4 · p. 7

6. Conclusion

• Pix2emb represents locations as embeddings that support outputs such as bounding boxes and segmentation masks.The conclusion describes it as a location modeling method.Source: §6 · p. 7
• NExT-Chat handles visual grounding, region captioning, grounded captioning, and complex question reasoning.The paper reports exploratory experiments demonstrating the method's effectiveness.Source: §6 · p. 7
• Future directions include improving detection and segmentation and extending NExT-Chat to multimodal agents that handle complex region-understanding tasks.Source: §6 · p. 7

7. Limitation

• The training data is primarily single-image and lacks diverse domains, limiting multiple-image handling and prediction accuracy on medical and satellite images.Source: §7 · p. 7