PaperlayerView exact revision on arXiv

Source-linked AI summary

LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark

Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Lu Sheng, Lei Bai, Xiaoshui Huang, Zhiyong Wang, Jing Shao, Wanli Ouyang

arXiv:2306.06687v3cs.CV

TL;DR

LAMM addresses limited transparency and insufficient 2D/3D coverage in multimodal language-model research by releasing an open-source dataset, benchmark, and extensible framework. It evaluates diverse vision tasks and reports that its baseline surpasses existing multimodal language models on image-related downstream tasks, while identifying limitations in GPT-API-generated data and benchmark stability.

  • Problem

    Prior multimodal models such as GPT-4V and Bard demonstrate visual capabilities, but their limited transparency leaves insufficient support for academic research.

  • Method

    LAMM combines image-and-point-cloud instruction-tuning data, a quantitative 2D/3D benchmark, and a modality-extensible MLLM framework with modality-specific training components.

  • Results

    LAMM’s baseline surpasses existing multimodal language models on image-related downstream tasks and is evaluated through broad experiments covering diverse vision tasks.

  • Takeaways & Limitations

    LAMM provides an open-source ecosystem for training and evaluating MLLMs across images and point clouds, supporting research on multimodal instruction tuning.

  • Takeaways & Limitations

    GPT-API-generated data may miss visual detail, contain factually incorrect or biased responses, and struggle with complex or ambiguous queries.

Abstract

from arXiv · show

Large language models have emerged as a promising approach towards achieving general-purpose AI agents. The thriving open-source LLM community has greatly accelerated the development of agents that support human-machine dialogue interaction through natural language processing. However, human interaction with the world extends beyond only text as a modality, and other modalities such as vision are also crucial. Recent works on multi-modal large language models, such as GPT-4V and Bard, have demonstrated their effectiveness in handling visual modalities. However, the transparency of these works is limited and insufficient to support academic research. To the best of our knowledge, we present one of the very first open-source endeavors in the field, LAMM, encompassing a Language-Assisted Multi-Modal instruction tuning dataset, framework, and benchmark. Our aim is to establish LAMM as a growing ecosystem for training and evaluating MLLMs, with a specific focus on facilitating AI agents capable of bridging the gap between ideas and execution, thereby enabling seamless human-AI interaction. Our main contribution is three-fold: 1) We present a comprehensive dataset and benchmark, which cover a wide range of vision tasks for 2D and 3D vision. Extensive experiments validate the effectiveness of our dataset and benchmark. 2) We outline the detailed methodology of constructing multi-modal instruction tuning datasets and benchmarks for MLLMs, enabling rapid scaling and extension of MLLM research to diverse domains, tasks, and modalities. 3) We provide a primary but potential MLLM training framework optimized for modality extension. We also provide baseline models, comprehensive experimental observations, and analysis to accelerate future research. Our baseline model is trained within 24 A100 GPU hours, framework supports training with V100 and RTX3090 is available thanks to the open-source society.

1 Introduction

LAMM addresses the limited transparency and modality coverage of prior multimodal language models by releasing an open-source dataset, benchmark, and extensible training framework for 2D and 3D vision.

  • Contributions: LAMM combines an instruction-tuning dataset, benchmark, and extensible MLLM training framework for multimodal AI-agent research.The framework separates modality-specific encoder, projector, and LLM fine-tuning blocks to reduce modality conflicts and support additional modalities.
  • Experimental scope: LAMM reports over 200 experiments analyzing MLLM capabilities and limitations, with its baseline surpassing existing multimodal language models on image-related downstream tasks.
  • Dataset: The dataset contains 186,098 image-language and 10,262 point-cloud-language instruction-response pairs generated from public datasets using GPT-API self-instruction.
  • Benchmark: The benchmark evaluates 9 image tasks across 11 datasets with over 62,439 samples and 3 point-cloud tasks across 3 datasets with over 12,788 samples.

2 Related Work

Prior multimodal language-model research follows system-integration or end-to-end trainable design, while instruction tuning improves language-model instruction following and downstream performance.

  • Multimodal large language models: System-design approaches connect LLMs with specialized vision or expert models, including Visual ChatGPT, MMREACT, ViperGPT, and HuggingGPT.
  • Multimodal large language models: End-to-end multimodal models unify language and vision through architectures such as Flamingo, BLIP-2, Kosmos, and PaLM-E.
  • Instruction tuning: Instruction tuning improves large language models’ ability to follow instructions and enhances downstream task performance.

3 Dataset

LAMM constructs a broad image-and-point-cloud instruction-tuning dataset from public data, using GPT-API prompts and templates to represent diverse vision tasks in language interactions.

  • Dataset construction: LAMM includes image and point-cloud data from public datasets, with GPT-API and self-instruction methods producing high-quality instructions and responses.The dataset contains 186K language-image and 10K language-3D instruction-response pairs.
  • Instruction formats: Four interaction formats cover daily dialogue, factual-knowledge dialogue, detailed description, and visual-task dialogue.
  • Task coverage: The dataset spans natural-language tasks such as captioning, scene-graph recognition, and VQA, alongside label-, box-, digit-, and word-output tasks.
  • Template generation: For visual-task dialogue, GPT-API-generated instruction-response templates accept ground-truth annotations for detection, OCR, counting, and related tasks.

4 Benchmark

LAMM introduces a quantitative benchmark for 2D and 3D MLLM tasks, standardizing inference outputs and combining task-specific, localization, and GPT-based evaluation.

  • Benchmark scope: The benchmark quantitatively evaluates MLLMs across diverse 2D and 3D vision tasks rather than relying only on demonstrations or user studies.The 2D benchmark covers 9 common image tasks across 11 datasets with over 62,439 samples.
  • Inference pipeline: Inference instructions specify task definitions, required output structures, and query questions before entity extraction supports fair evaluation.
  • Evaluation metrics: Evaluation combines traditional task-specific metrics, a binary locating metric, and a GPT metric for answer relevance and accuracy.The GPT metric scores accuracy, relevance, fluency, logical coherence, and information richness.
  • Evaluation settings: The benchmark supports both zero-shot evaluation and fine-tuning evaluation for mainstream 2D and 3D tasks.

5 Experiments and Results

Experiments evaluate LAMM across 2D and 3D vision tasks using traditional, binary locating, and GPT-based metrics. Results show stronger performance after fine-tuning and improved localization, while precise spatial prediction, domain-shifted data, and some captioning metrics remain challenging.

  • Framework: The baseline processes image and point-cloud inputs through modality-specific encoders and projectors before concatenating vision and text tokens for LLM training.LoRA parameters are added to self-attention projection layers, with separate parameters for different vision modalities.
  • Zero-shot 2D vision tasks: Zero-shot MLLMs recognize open-vocabulary classes and answer some visual questions, but perform poorly on object detection, counting, keypoint localization, captioning, and fine-grained classification.The largest weaknesses concern spatial outputs and exact object counts or keypoint positions.
  • Fine-tuning on image tasks: 91% accuracy on CIFAR10 and 13% mAP on VOC2012 after fine-tuning demonstrate improved classification and localization compared with 4.8% zero-shot VOC2012 mAP.The results indicate that detection-data fine-tuning gives the baseline object-localization ability.
  • Point-cloud tasks: Fine-tuning significantly improves all tested 3D scene-understanding tasks, with nearly 100% ScanQA multiple-choice accuracy but a possible overfitting issue.The authors attribute the concern to the narrow training/test gap and the small scale of the 3D dataset.
  • Binary locating and GPT metrics: Detection instructions significantly improve binary localization accuracy, while LAMM’s GPT Metric score remains below LLaVA but exceeds MiniGPT4 and mPLUG-owl by a wide margin.The GPT Metric covers VQA, classification, captioning, and smaller numbers of detection and counting tasks.
  • Observation and analysis: Instruction and reasoning prompts improve VQA, whereas domain-shifted data and precise bounding-box prediction remain difficult; GPT-based caption evaluation may be more suitable than BLEU.LAMM produces detailed captions but has low correlation with ground-truth wording, lowering BLEU scores.

6 Limitations

LAMM identifies limitations in its GPT-generated data, benchmark metrics, and baseline framework. The authors note that GPT-generated text may omit visual details or contain factual errors, benchmark results may be unstable, and the framework remains open to further development.

  • Dataset: GPT-API relies on textual context rather than direct visual information, which may omit image details and produce plausible but factually incorrect responses.The authors also report difficulties with complex or ambiguous queries and possible inherited biases.
  • Dataset: Manual sampling was used to check generated data against societal values, privacy, security, toxicity, and fairness requirements.The authors also provide prompts and additional quality evaluations for transparency.
  • Benchmark: LAMM’s benchmark metrics may fluctuate across experiments because language-model outputs are diverse.The benchmark uses GPT-eval and binary localization as initial evaluation strategies.
  • Benchmark: The authors state that more appropriate metrics and greater benchmark-result stability require further research.This is identified as a direction for future investigation.
  • Framework: The framework is described as a simple baseline framework, leaving room for more careful MLLM design to improve capabilities and performance.The limitation concerns the framework’s current scope rather than a reported failure of the baseline.

7 Conclusion

The paper presents LAMM as an open-source ecosystem for training and evaluating multimodal large language models across images and point clouds. It releases datasets, benchmarks, code, and an extensible framework to support further MLLM research and modality extension.

  • Conclusion: LAMM introduces open-source image and point-cloud instruction-tuning datasets and a benchmark for training and evaluating MLLMs.The conclusion frames these resources as an ecosystem for multimodal research.
  • Conclusion: The framework is designed to facilitate extending MLLMs to additional modalities.The paper presents this as part of its effort to support broader multimodal-agent research.
  • Conclusion: The authors report that MLLMs can handle visual modalities, including images and point clouds, with potential for generalization through instruction tuning.This conclusion is stated within the scope of the paper’s experiments.
  • Conclusion: The codebase, baseline model, instruction-tuning dataset, and evaluation benchmark are publicly available to foster an open research community.The appendix identifies the public repository and reports training the framework in about 20 A100 GPU hours.

B Dataset

LAMM constructs multimodal instruction-tuning data for images and point clouds by combining GPT-generated dialogues and descriptions, factual knowledge, and converted computer-vision task annotations. The resulting datasets emphasize fine-grained information, factual knowledge, and coverage of diverse 2D and 3D tasks.

  • Image Instruction Tuning Dataset: The 2D dataset combines daily dialogue, detailed description, factual-knowledge dialogue, and visual-task dialogue samples.Its construction uses GPT-API, visual relationships, fine-grained categories, Wikipedia-derived facts, and converted vision-task annotations.
  • Image Instruction Tuning Dataset: 42K classes of Wikipedia knowledge-graph facts from Bamboo provide content for factual-knowledge dialogues and broader common-sense coverage.QID labels and corresponding Wikipedia descriptions are used to generate instruction-tuning data with GPT-API.
  • Image Instruction Tuning Dataset: Traditional computer-vision annotations are converted into natural-language dialogues so MLLMs can perform tasks through language interaction.The included tasks cover classification, detection, keypoint detection, OCR, and object counting.
  • Image Instruction Tuning Dataset: 186K unique language-image instruction-following samples comprise 49K daily dialogues, 49K detailed descriptions, 42K factual-knowledge dialogues, and 46K visual-task dialogues.The dataset is described as incorporating fine-grained and dense information from object relationships and factual-knowledge sources.
  • Point Cloud Instruction Tuning Dataset: The point-cloud dataset uses GPT-generated conversations and descriptions alongside visual-task dialogues converted from 3D vision annotations.Point clouds come from 3RScan, while CLEVR3D VQA annotations are converted into declarative captions and 3DSSG supplies object attributes and relationships.
  • Data Quality: The construction process uses cold-start generation, manual checks, prompt adjustment, and subsequent random checks to improve data quality and alignment with inputs.The authors describe these checks as addressing quality and ethical concerns.
  • Data Quality: The dataset combines publicly available CC-BY data with GPT-generated language, while the authors acknowledge that problematic content may remain and plan to obfuscate faces and license plates.The stated social-impact measures emphasize licensing, privacy, transparency, and content review.

C.1 Benchmark on image tasks

The image benchmark evaluates MLLMs across nine computer-vision tasks using task-specific outputs and metrics, with zero-shot and fine-tuning settings. It also introduces binary locating and GPT-based evaluation strategies for localization and multimodal question answering.

  • The benchmark covers nine image tasks spanning classification, detection, pose estimation, visual question answering, facial classification, OCR, counting, and related vision capabilities.
  • Task definitions: CIFAR-10 evaluates open-vocabulary classification using noun extraction and synonym expansion, while VOC2012 evaluates detection with extracted categories and bounding boxes for mAP50.
  • Task definitions: SQAimage and AI2D test image question answering, Flickr30k uses BLEU-4 for captioning, and UCMerced Land Use reports accuracy for fine-grained classification.
  • Task definitions: FSC147 measures object counting with mean absolute error, SVT measures OCR with Word Accuracy, CelebA evaluates facial attributes, and LSP keypoint detection uses PCK.
  • Inference design: System messages specify task definitions and output structures, and some image tasks additionally prompt separate reasoning and final-answer responses.
  • Evaluation settings: The benchmark provides zero-shot evaluation on tasks without training-data overlap and fine-tuning evaluation on mainstream detection, classification, and VQA datasets.
  • Evaluation metrics: Binary Locating Metric tests whether a predicted object position falls inside its ground-truth box, while GPT Metric ranks answers by relevance and accuracy.

D Implementation Details

The implementation formats multimodal examples with task-specific system prompts and vision delimiters, while training separates 2D and 3D models and optimizes lightweight parameters. Qualitative examples reveal both fine-grained recognition and descriptive detail beyond reference captions.

  • 2D and 3D models are trained independently, while the LLM can be shared and only feature projection and LoRA parameters are optimized.
  • Optimization: Training uses Adam with an initial learning rate of 5e-4, linear decay, two epochs for 2D experiments, and 10,000 iterations for 3D experiments on four A100-80GB GPUs.
  • Optimization: The 2D experiments last about 8 hours for 186K samples, while 3D experiments require about 3 hours.
  • Data formatting: Multimodal data uses system, query, and response fields with modality-specific < Img > and < Pcl > delimiters, trained using next-token prediction on response text.
  • Qualitative results: CIFAR10 examples judged incorrect can contain granular labels whose reference objects visually resemble the input images.
  • Qualitative results: Flickr30k responses receive poor BLEU scores when additional relevant image details are absent from the ground-truth captions.

E.3 Comparison with LLaVA on detection and counting tasks

The comparison examines models trained on the LAMM dataset against LLaVA for object detection and counting. LAMM’s model provides more accurate localization outputs and better counting performance after fine-tuning on detection-related data.

  • The comparison uses visualizations of ground-truth and extracted response bounding boxes for object detection, alongside counting results on FSC147.
  • LLaVA identifies approximate object locations but not precise bounding-box coordinates, whereas the LAMM-trained model provides more accurate coordinates after detection-related fine-tuning.
  • The LAMM-trained model also exhibits better counting performance, and the paper connects counting with localization because it requires locating objects.

E.4 Results of binary-loc metric and GPT metric

The paper compares LAMM-trained models with LLaVA using binary locating and GPT-based evaluation. The LAMM-trained model shows more precise localization and receives higher GPT rankings for more specific and accurate descriptions.

  • Binary locating: On the binary locating metric, the LAMM-trained model demonstrates more precise localization than LLaVA.
  • Binary locating: The model’s predicted key points are visualized, and bounding-box outputs are decomposed into top-left and bottom-right coordinates during entity extraction.
  • GPT metric: The GPT metric ranks the LAMM-trained model’s image-captioning responses higher than LLaVA because they are judged more specific and accurate.
  • Overall comparison: These results support the model’s performance in detecting, locating, and describing objects in images.
Loading 2306.06687v3…