Source-linked AI summary
LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, Jianfeng Gao
TL;DR
Biomedical vision-language models remain limited in open-ended conversation about biomedical images. LLaVA-Med addresses this gap with GPT-4-generated instruction data and two-stage curriculum training, outperforming supervised state of the art on selected biomedical VQA metrics.
Problem
General-domain visual assistants and classification-based biomedical VQA systems remain limited for open-ended conversational understanding of biomedical images.
Method
LLaVA-Med uses GPT-4 to generate biomedical instruction-following data and adapts a general-domain vision-language model through two-stage curriculum training.
Results
LLaVA-Med outperforms supervised state of the art on closed-set VQA-RAD and PathVQA and achieves state of the art on open-set SLAKE.
Takeaways & Limitations
The work provides a biomedical multimodal conversational assistant with domain knowledge and strong chat abilities.
Takeaways & Limitations
LLaVA-Med remains limited by hallucinations and weak in-depth reasoning.
Abstract
from arXiv · showhide
Conversational generative AI has demonstrated remarkable promise for empowering biomedical practitioners, but current investigations focus on unimodal text. Multimodal conversational AI has seen rapid progress by leveraging billions of image-text pairs from the public web, but such general-domain vision-language models still lack sophistication in understanding and conversing about biomedical images. In this paper, we propose a cost-efficient approach for training a vision-language conversational assistant that can answer open-ended research questions of biomedical images. The key idea is to leverage a large-scale, broad-coverage biomedical figure-caption dataset extracted from PubMed Central, use GPT-4 to self-instruct open-ended instruction-following data from the captions, and then fine-tune a large general-domain vision-language model using a novel curriculum learning method. Specifically, the model first learns to align biomedical vocabulary using the figure-caption pairs as is, then learns to master open-ended conversational semantics using GPT-4 generated instruction-following data, broadly mimicking how a layperson gradually acquires biomedical knowledge. This enables us to train a Large Language and Vision Assistant for BioMedicine (LLaVA-Med) in less than 15 hours (with eight A100s). LLaVA-Med exhibits excellent multimodal conversational capability and can follow open-ended instruction to assist with inquiries about a biomedical image. On three standard biomedical visual question answering datasets, LLaVA-Med outperforms previous supervised state-of-the-art on certain metrics. To facilitate biomedical multimodal research, we will release our instruction-following data and the LLaVA-Med model.
1 Introduction
General-domain multimodal assistants are less effective for biomedical images, while prior biomedical VQA methods are poorly suited to open-ended instruction-following. LLaVA-Med extends multimodal instruction-tuning to biomedical conversations and introduces an automatically generated biomedical visual instruction-following dataset.
- Biomedical image-text pairs differ substantially from general web content, causing general-domain visual assistants to avoid biomedical questions or produce incorrect responses and hallucinations.
- Prior biomedical visual question answering methods typically classify among training-set answers rather than support open-ended instruction-following.
- LLaVA-Med is presented as a first attempt to extend multimodal instruction-tuning to biomedical images for end-to-end biomedical multimodal conversation.
- The data pipeline samples biomedical image-text pairs from PMC-15M and uses GPT-4 to generate instructions from text alone, with the text serving as the intended output.The pipeline requires zero manual annotations and produces diverse (image, instruction, output) instances.
2 Related Work
Prior biomedical chatbots primarily adapt open-source LLMs with customized biomedical instruction-following data, while Visual Med-Alpaca is identified as the only existing multimodal biomedical chatbot accepting image inputs. Related biomedical VQA work distinguishes discriminative classification methods from generative approaches, and LLaVA-Med differs architecturally from prefix-tuning and Visual Med-Alpaca systems.
- Biomedical Chatbots: Several biomedical chatbots fine-tune open-source LLMs on customized biomedical instruction-following datasets.Examples include ChatDoctor, Med-Alpaca, PMC-LLaMA, Clinical Camel, DoctorGLM, and Huatuo.
- Biomedical Chatbots: Visual Med-Alpaca is described as the only existing multimodal biomedical chatbot that accepts image inputs.Both Visual Med-Alpaca and LLaVA-Med use similar input-output data formats.
- Biomedical Chatbots: LLaVA-Med is an end-to-end neural model, whereas Visual Med-Alpaca connects multiple image-captioning models to an LLM and uses a classifier to select the responsible model.The classifier determines whether, or which, biomedical captioning model handles the image.
- Biomedical Visual Question Answering: Biomedical visual question answering methods are categorized as discriminative or generative, with discriminative methods treating VQA as classification over a predefined answer set.The passage notes that discriminative methods achieve good performance but operate in a closed set.
- Model Architecture: Compared with prior prefix tuning, LLaVA-Med uses a linear projection and a 7B language model rather than a three-layer MLP and language models ranging from 1.5B to 2.7B.The prior approach connects frozen image encoders and causal language models through a trainable module.
3 Biomedical Visual Instruction-Following Data
The paper introduces a biomedical visual instruction-following dataset built from existing biomedical image-text pairs through machine-human co-curation. It contains concept-alignment and instruction-following data for separate training stages, with GPT-4 generating diverse multi-round conversations from captions.
- Dataset overview: The dataset addresses the lack of multimodal biomedical instruction-following resources and comprises concept-alignment and instruction-following sets for different training stages.The data are created from widely existing biomedical image-text pairs through machine-human co-curation.
- Biomedical Concept Alignment Data: 600K image-text pairs are sampled from PMC-15M, using a 30-word caption-length cutoff to select concise versus detailed description questions.Captions shorter than 30 words account for 25% of PMC-15M and determine which question list is used.
- Biomedical Instruction-Tuning Data: GPT-4 generates diverse multi-round biomedical image conversations from captions by producing questions and answers in a tone as if it could see the image.GPT-4 is prompted with text only, rather than the image itself.
- Biomedical Instruction-Tuning Data: Three instruction-data variants—60K-IM, 60K, and 10K—are created to ablate inline-mention context, dataset size, and self-instruction generation strategies.The variants are used to study their impact on trained LLaVA-Med in experiments.
4 Adapting Multimodal Conversational Models to the Biomedical Domain
LLaVA-Med adapts the general-domain LLaVA to biomedicine through two-stage curriculum training, followed by optional fine-tuning on biomedical VQA datasets. The procedure aligns biomedical visual concepts before end-to-end conversational instruction tuning while keeping development affordable.
- Two-stage adaptation: LLaVA-Med starts from general-domain LLaVA and uses the same architecture, with a linear projection layer connecting the vision encoder and language model.Training follows a two-stage procedure.
- Biomedical concept feature alignment: Stage 1 filters PMC-15M to 600K image-text pairs and trains image-description instruction data to align biomedical visual concepts with textual word embeddings.The visual encoder and language-model weights remain frozen while only the projection layer is updated.
- End-to-end instruction-tuning: Stage 2 freezes the visual encoder while updating the projection layer and language model on biomedical language-image instruction-following data for conversational task completion.This stage develops a biomedical chatbot through end-to-end instruction tuning.
- Fine-tuning to downstream datasets: After the two training stages, LLaVA-Med is fine-tuned on three biomedical VQA datasets spanning varied dataset sizes and diverse biomedical subjects.The datasets support free-form responses to close-set and open-set questions, with candidate answers supplied for close-set questions.
- Discussion: 7 and 8 hours are required for stages 1 and 2, respectively, on 8 40G A100 GPUs.The paper presents this as an affordable development-cost property of LLaVA-Med.
5 Experiments
Experiments evaluate LLaVA-Med’s biomedical conversational ability and benchmark performance using language-only GPT-4 evaluation and three biomedical VQA datasets. The full two-stage model consistently improves over general-domain LLaVA, surpasses supervised state of the art on selected closed-set tasks, and transfers zero-shot to Chinese questions.
- 5 Experiments: Experiments assess multimodal biomedical instruction following and benchmark performance, using only language-only GPT-4 throughout evaluation.The evaluation covers open-ended visual-chatbot capability and comparison with existing methods on standard benchmarks.
- Biomedical visual conversation: The conversational evaluation contains 193 novel questions generated from 50 unseen PMC-15M image-caption pairs, covering conversation and detailed-description questions.GPT-4 scores candidate responses for helpfulness, relevance, accuracy, and detail against a reference prediction.
- Biomedical visual conversation: Full two-stage LLaVA-Med consistently outperforms general-domain LLaVA, while increasing instruction data from 10K to 60K samples improves performance.Stage-1-only training improves biomedical concept coverage but is insufficient for diverse chatbot instruction following; 60K-IM yields a slight further improvement.
- Biomedical visual conversation: LLaVA-Med precisely answers biomedical visual conversations, whereas LLaVA often hallucinates from commonsense; language-only GPT-4 serves as an upper-bound comparison.GPT-4 receives golden captions and inline mentions rather than interpreting the raw image.
- Biomedical VQA benchmarks: LLaVA-Med is evaluated on three biomedical VQA datasets, and all variants outperform LLaVA while fine-tuning exceeds supervised state of the art on closed-set VQA-RAD and PathVQA.Closed-set questions use accuracy, while open-set questions use recall of ground-truth tokens in generated sequences.
- Case Study I: Zero-shot on Chinese Questions: Despite no Chinese instruction-following data, the 60K-IM model correctly answers Chinese SLAKE questions in zero-shot transfer, while existing models fail cross-language transfer.The authors attribute this ability probably to multilingual knowledge learned in LLaMA/Vicuna.
6 Conclusions
LLaVA-Med is a biomedical language-and-vision model built with a GPT-4-based self-instruct data curation pipeline, showing strong domain chat abilities and improved VQA performance. However, it remains limited by hallucinations and weak in-depth reasoning, motivating future work on quality and reliability.
- Contributions: LLaVA-Med is a biomedical large language-and-vision model trained using a high-quality language-image instruction-following dataset curated through language-only GPT-4 and external knowledge.The dataset was created with a self-instruct approach and data curation pipeline.
- Contributions: LLaVA-Med demonstrates strong domain-knowledge chat abilities and, after subsequent fine-tuning, outperforms previous supervised SoTA on three VQA datasets on certain metrics.The conclusion reports these gains without specifying the individual metrics or values.
- Limitations and future work: LLaVA-Med is limited by hallucinations and weak in-depth reasoning, limitations common to many large multimodal models.Future work is directed toward improving the model’s quality and reliability.
A Data
The data use instruction sets for briefly and thoroughly describing biomedical image content. Each set preserves the same meaning across naturally varied phrasings and is documented in a separate table.
- Brief image description: The brief image-description instructions express the same meaning using natural-language variation.They are listed in Table 7.
- Detailed image description: The detailed image-description instructions similarly vary the wording while preserving meaning.They are listed in Table 8.
B Prompts
The section describes prompting GPT-4 to generate medical visual instruction-following data for biomedical topics. The prompt uses manually curated few-shot examples with figure-caption inputs and responses, including contextual information from the corresponding PubMed paper.
- Prompt design: GPT-4 is prompted to generate medical visual instruction-following data.The system prompt specifies an AI assistant specialized in biomedical topics.
- Prompt design: The prompt includes manually curated few-shot examples pairing each input sample’s context with an output response.These examples are included to guide construction of the medical visual instruction-following data.
- Few-shot example: Example inputs include figure captions such as a chest X-ray describing cardiomegaly, bilateral interstitial infiltrates, and right-sided pleural effusion.The example caption is presented as the figure-caption input to the prompt.
- Few-shot example: Figure context adds sentences from the corresponding PubMed paper that mention the figure.This provides additional context alongside the figure caption in the prompt’s example input.