u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model

• Despite strong general-purpose MLLM understanding, regional and pixel-level capabilities remain less remarkable, while differing downstream objectives make homogeneity challenging.Regional methods can require extensive training data, while pixel-level methods require specific segmentation modules.Source: §1 · p. 1
• u-LLaVA aligns image and spatio-temporal features, then jointly trains task-specific projectors and decoders for region and pixel-level understanding.The framework also describes image and video features for modality alignment and makes additional downstream modules patchable.Source: §1 · p. 1, Figure 1 · p. 2
• A projector connects the MLLM to SAM to exchange semantic and pixel-level capabilities, while an independent location decoder reduces regional training-data needs.The method also uses task-related prompt pools and introduces ullava-277K, a mask-based, region-specific dataset, with missing annotations supplemented by GPT-3.5.Source: §1 · p. 1
• The contributions include image- and video-based modality alignment, joint instruction tuning across levels, and public release of the model, code, and ullava-277K dataset.The authors report comprehensive experiments demonstrating the method's effectiveness.Source: §1 · p. 2

2.1 MLLMs

• MLLMs have demonstrated impressive image-level understanding but show limited capabilities on pixel- or region-level tasks.Source: §2.1 · p. 2

2.2 Region-Level Comprehension

• Referring expression comprehension is a regional task with established datasets, and approaches include pix2seq, regional encoding-decoding, and external modules.Source: §2.2 · p. 2

2.3 Pixel-Level Understanding

• Progress on mask-aware tasks such as referring expression and salient object segmentation has been relatively slow, with methods using grounding outputs as SAM input or specific end-to-end decoders.Source: §2.3 · p. 2

3.1 Efficient Visual Alignment

• Stage I aligns image/video and text through CLIP and a visual projector with Vicuna, adding spatial-temporal video representations and two special tokens.The coarse-grained objective maximizes likelihood over conversation tokens conditioned on visual embeddings and context.Source: §3.1 · pp. 2–3

3.2 Joint Instruction Tuning

• Joint instruction tuning brings general, region, and pixel features into the same training stage, where many methods include only one or two aspects.Source: §3.2 · p. 3
• General-aware tuning retains the model structure and emphasizes multi-turn dialogue and complex reasoning; mask-aware tuning maps mask-token hidden states into SAM text embeddings.Source: §3.2 · p. 3
• Region-aware tuning maps location-token states to target coordinates, while segmentation annotations can supply bounding boxes when detection labels are absent.The fine-grained objective combines mask-prediction loss with target-box prediction loss.Source: §3.2 · p. 3

3.3 Dataset Construction

• The authors reorganize public datasets; for Salient-15K, mask-extracted objects are described by BLIP2, then GPT-4o phrases tags into templates.Table 3 indicates which training datasets are used in Stage I versus Stage II.Source: §3.3 · p. 3, Table 3 · p. 3

4.1 Implementation Details

• Experiments use Vicuna v1.1 and CLIP ViT-L/14, with 8 NVIDIA Tesla A100 80G GPUs and PyTorch.AdamW is used with zero weight decay; learning rates are 2e-3 and 2e-5, or 2e-4 with LoRA.Source: §4.1 · p. 3

4.2 Evaluation Metrics

• cIoU measures total intersection pixels divided by total union pixels for pixel segmentation.Some works also call this metric overall-IoU (oIoU).Source: §4.2 · p. 3
• Region grounding uses Precision@X, the percentage of samples whose IoU exceeds threshold X; this work sets X to 0.5.Source: §4.2 · pp. 3–4
• Table 4 reports RES results with cIoU, distinguishes specialist CV models from MLLMs, and marks LoRA-trained MLLMs with ⋆.The top two results are bolded and underlined.Source: Table 4 · p. 4

• 9.28 average improvement in the cIoU indicator over LISA-7B∗(ft); u-LLaVA-7B also surpasses UNINEXT(H) on three benchmarks using one-tenth the training data.Source: §4.3 · p. 4

4.4 Pixel-level Intent Understanding Performance

• u-LLaVA achieves state-of-the-art performance on DUT-OMRON, DUTS-TE, and ECSSD salient-segmentation benchmarks.The datasets contain 5,168, 5,019, and 1,000 test images, respectively.Source: §4.4 · p. 4

4.5 Region-Level Understanding Performance

• On REC, u-LLaVA outperforms MLLMs such as Shikra using one-tenth the data, but remains behind UNINEXT(H), partly due to input resolution and task interference.Source: §4.5 · p. 4
• The region-level inference workflow combines region-decoder output and the generated mask into a final regression box; grounding models and generated tags can add redundancy.Source: §4.5 · p. 4, Figure 2 · p. 4

• u-LLaVA-1.5 ranks first on ScienceQA-IMG with Vicuna-7B-v1.1 and second to LLaVA-1.5 on MMBench-Test and GQA.Source: §4.6 · pp. 4–5
• The model increases input resolution to 336 for these tasks, which the paper notes are not its primary focus.Source: §4.6 · p. 5

4.7 Dataset Ablation

• Greater diversity in Stage II training data improves generalization and robustness while reducing the risk of task-specific overfitting.The authors report these findings for the dataset ablation in Table 8.Source: §4.7 · p. 5

4.8 Qualitative examples

• Qualitative grounding and segmentation comparisons with LISA, Shikra, and CogVLM are presented in Figure 3.The passage identifies the comparison tasks and methods but does not state which method performs best.Source: §4.8 · p. 5
• Figures 4 and 5 provide additional conversation illustrations.The passage identifies these as further examples, without describing their outcomes.Source: §4.8 · p. 5

5 Conclusions

• u-LLaVA jointly tunes instructions at the global, regional, and pixel levels.The conclusion presents this multi-level instruction tuning as the model's defining approach.Source: §5 · p. 5
• The authors report optimal performance across various comprehension-based tasks.The conclusion does not specify benchmark names or numerical results in this passage.Source: §5 · p. 5
• The authors describe the study as exploratory and experimental, noting that MLLM pre-training and task adaptation remain open areas.They state that open-sourcing their work can assist development in the field.Source: §5 · p. 5

7.1 Templates

• Table 7 is captioned as comparing leading methods on popular multimodal benchmarks, with dagger-marked models trained for one epoch for fair comparison.The caption says the top two results are outlined in bold and underline; it does not provide the table's values.Source: Table 7 · p. 6
• Figures 4 and 5 show conversation and video-captioning examples, while Figure 6 shows image inpainting and object-removal examples.These captions identify example outputs for u-LLaVA-7B.Source: Figure 4 · p. 6, Figure 5 · p. 6, Figure 6 · p. 6
• The appendix shows task templates for salient-object segmentation, video captioning, and referring-expression segmentation (RES).Examples include prompts to identify or segment salient objects, describe videos, and locate or mask a specified class.Source: §7.1 · p. 6
• Table 8 is captioned as an ablation of Stage II training datasets using cIoU as the performance indicator.The caption identifies the ablation scope and metric, without giving the table's values.Source: Table 8 · p. 7

7.2 Construction of Salient-15K

• For Salient-15K, the authors extract principal subjects from MSRA-10K and MSRA-B images, use BLIP2 for initial descriptions, then GPT3.5 to parse target labels and expand descriptions.Figure 7 is identified as the dataset's process workflow.Source: §7.2 · p. 7, Figure 7 · p. 7