PaperlayerView exact revision on arXiv

Source-linked AI summary

MoleculeQA: A Dataset to Evaluate Factual Accuracy in Molecular Comprehension

Xingyu Lu, He Cao, Zijing Liu, Shengyuan Bai, Leqing Chen, Yuan Yao, Hai-Tao Zheng, Yu Li

arXiv:2403.08192v1cs.CLq-bio.BM

TL;DR

Molecular LLMs can generate false statements, while existing metrics may not establish the factual accuracy of their molecular descriptions. The paper constructs MoleculeQA, a taxonomy-guided QA benchmark, and uses it to evaluate molecular models, finding persistent comprehension deficiencies and identifying important factors for molecular understanding.

  • Problem

    Molecular LLMs may generate false statements, and expert review of hallucinations in molecular text is difficult and costly.

    Source: §1 · pp. 1–2

  • Method

    MoleculeQA uses a molecular-domain taxonomy to convert authoritative descriptions into QA pairs for factual evaluation.

    Source: §1 · p. 2

  • Results

    Evaluation identifies comprehension deficiencies and important factors in existing models; none exceeds 50% accuracy on Property or Application.

    Source: §1 · p. 2, §4.3 · p. 6

  • Takeaways & Limitations

    The evaluation identifies shortcomings and provides guidance for future development of molecular LLMs.

    Source: §6 · p. 8

    The data distribution is imbalanced, with Structure and Source dominating; resource constraints also prevented full fine-tuning of models with 7B parameters or more.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models are playing an increasingly significant role in molecular research, yet existing models often generate erroneous information, posing challenges to accurate molecular comprehension. Traditional evaluation metrics for generated content fail to assess a model's accuracy in molecular understanding. To rectify the absence of factual evaluation, we present MoleculeQA, a novel question answering (QA) dataset which possesses 62K QA pairs over 23K molecules. Each QA pair, composed of a manual question, a positive option and three negative options, has consistent semantics with a molecular description from authoritative molecular corpus. MoleculeQA is not only the first benchmark for molecular factual bias evaluation but also the largest QA dataset for molecular research. A comprehensive evaluation on MoleculeQA for existing molecular LLMs exposes their deficiencies in specific areas and pinpoints several particularly crucial factors for molecular understanding.

1 Introduction

Molecular LLMs can produce fluent but factually false descriptions, while common lexical metrics do not establish factual reliability. MoleculeQA addresses this gap with a taxonomy-guided QA benchmark and an evaluation of existing models.

1 Introduction

  • Existing molecular LLMs can generate false statements, and lexical-similarity metrics may miss factual errors in otherwise fluent descriptions.Table 1 illustrates factual errors in BioT5-base outputs when compared with the ground truth.Source: §1 · pp. 1–2, Table 1 · p. 1
  • Detecting hallucinations in molecular text requires expertise and professional knowledge, making human review difficult and costly.The paper argues that quantifying models’ molecule knowledge is valuable in light of these detection costs.Source: §1 · pp. 1–2
  • MoleculeQA evaluates molecular knowledge through questions covering properties, sources, structures, and applications.Its taxonomy-guided construction converts authoritative molecular descriptions into QA pairs at different levels of granularity.Source: §1 · p. 2
  • The authors identify MoleculeQA as the first factual-evaluation benchmark in the molecular domain and the largest molecular QA dataset.Its construction uses a hierarchical domain taxonomy and converts molecular descriptions into QA pairs.Source: §1 · p. 2
  • Evaluation on MoleculeQA reveals deficiencies in existing molecular models and identifies factors considered crucial for molecular understanding.The authors report testing a series of models in the molecule domain.Source: §1 · p. 2

2 Related Work

Prior work evaluates molecular LLMs through generation and retrieval tasks, but these approaches leave factual accuracy and fine-grained text–structure connections insufficiently assessed. The paper positions MoleculeQA as a taxonomy-based QA benchmark amid limited comprehensive molecular QA datasets.

2.1 Molecule Understanding LLMs

  • Molecular LLM evaluation includes generation tasks such as captioning and de novo molecule generation, and contrastive tasks such as cross-modal retrieval.The cited examples include generative models MolT5 and BioT5, and contrastive models MoMu and MoleculeSTM.Source: §2.1 · p. 2
  • Captioning metrics measure lexical similarity rather than chemical knowledge or factual bias, while retrieval overlooks fine-grained links between text and substructures.MoleculeQA is presented as a hierarchical QA framework for evaluating molecular knowledge inference.Source: §2.1 · p. 2

2.2 Domain-Specific QA

  • Although domain-specific QA datasets span multiple formats and sources, comprehensive, diverse, high-quality QA datasets remain scarce in the molecular domain.The paper contrasts MoleculeQA’s taxonomy-derived descriptive-text questions with DrugChat’s feature and answer limitations and BioMedGPT’s inherited evaluation issues.Source: §2.2 · pp. 2–3

3 Method

The caption analysis finds lexical gains without discernible factual-reliability gains; the paper then describes taxonomy-guided QA construction and MoleculeQA’s coverage and scale.

3.1 Exposure of Factual Bias

  • 17.6% BLEU-2 improvement brought no discernible reliability gain; factual accuracy remained 0.4.Models often omitted application details and relevant properties, while structural descriptions showed discrepancies exceeding 63% from ground truth.Source: §3.1 · pp. 3–4, Figure 1 · p. 3

3.2 Domain Taxonomy Construction

  • The taxonomy combines ChEBI-20 with T3DB, FDA Pharm Classes, and DrugBank, then uses hybrid topic extraction and expert normalization to form 587 topics across three levels.The process includes rule-based and few-shot extraction, semantic aggregation, and expert clustering.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3 MoleculeQA Construction

  • MoleculeQA turns topic-linked descriptions into QA pairs, moving to a parent topic when source text lacks enough detail for a query.Annotators design questions; ChatGPT drafts positive options, while BioT5 selects semantically similar negatives after filtering synonyms and overlap. Molecules are split by scaffold, and human evaluation reports a test-split error rate below 1%.Source: §3.3 · pp. 4–5

3.4 Data Analysis

  • MoleculeQA is presented as the first molecular factual-knowledge benchmark and the largest molecular-field QA dataset, with taxonomy-topic coverage.The train, development, and test splits contain 49,993, 5,795, and 5,786 QA samples; Structure accounts for approximately two-thirds of the dataset.Source: §3.4 · pp. 5–6, Table 6 · p. 5

4 Experiment

The experiments benchmark molecular, general, and large-scale universal LLMs on MoleculeQA using full fine-tuning, LoRA, or few-shot prompting. Results show uneven accuracy across aspects and model families, with notable differences by training data, modality, and model scale.

4.1 Baseline Models

  • The benchmark includes molecular LLMs, general LLMs, and large-scale universal models; the latter were evaluated in few-shot settings.The model groups include systems trained on biomolecular data, general-purpose models, and API-accessed GPT models.Source: §4.1 · p. 6

4.2 Evaluation Setups

  • Evaluation used full fine-tuning, LoRA fine-tuning, or prompts with 10 QA examples sampled from each of four aspects.Accuracy is the ratio of correctly answered test samples; training approaches and hyperparameters followed the original papers.Source: §4.2 · p. 6

4.3 Main Results

  • Property and Application were difficult across models, with none exceeding 50% accuracy in either aspect.Source was generally the easiest aspect for each model.Source: §4.3 · p. 6, Table 8 · p. 7
  • Molecular LLMs achieved at least 51% total accuracy, while decoder-only general LLMs other than T5 remained at or below 50%.This pattern held for full fine-tuning and LoRA.Source: §4.3 · p. 6, Table 8 · p. 7
  • BioT5 exceeded T5 accuracy by 6.5%, while T5-base exceeded MolT5-base by 5.1% in total accuracy.The T5-versus-MolT5 ordering contrasts with their molecule-captioning performance.Source: §4.3 · p. 7, Table 8 · p. 7
  • Among 7B models with similar LoRA parameters, Pythia ranked above BLOOM, GALACTICA, and Llama; increasing size from 7B to 13B brought mild accuracy gains.BioMedGPT-7B performed best among Llama-based models, while Mol-Instruction slightly improved on Llama and Vicuna.Source: §4.3 · p. 7, Table 8 · p. 7
  • MoMu and MolCA, which combine molecular graphs and text, outperformed their respective single-modality base models, MolT5 and GALACTICA.In a 10-shot setting, GPT-4 achieved accuracy comparable to certain specialized models, while smaller models declined sharply.Source: §4.3 · p. 7, Table 8 · p. 7

5 Analysis

The analysis finds that molecular comprehension remains weak in important aspects and varies across topics. It identifies domain-corpus scale and diversity, modality modeling, and model scale as relevant factors.

5.1 In-depth Performance Analysis (RQ1)

  • Property and Application accuracy remained unsatisfactory, despite Source and Structure accuracy exceeding 40% across all plotted categories for T5-base and BioT5.Accuracy varied by topic, with Agricultural Chemical and Approval status notably weak.Source: §5.1 · pp. 7–8, Figure 5 · p. 8

5.2 Crucial Factor Attribution (RQ2)

  • BioT5 outperformed T5 and BioMedGPT outperformed models trained on smaller or more general corpora, highlighting the importance of large, diverse, high-quality molecular-domain datasets.The cited decoder-only ordering was BioMedGPT > Mol-Instruction > Vicuna > Llama.Source: §5.2 · p. 8
  • LoRA-based textual methods lagged multimodal-fusion or full-fine-tuning methods, while MolCA’s Q-Former graph adaptation improved more than MoMu’s linear-layer adaptation.The authors interpret the performance gap as underscoring the need for sufficient trainable parameters to model textual or graph modalities.Source: §5.2 · p. 8

5.3 Scaling Law for Molecular LLMs (RQ3)

  • Performance showed a pronounced scaling effect across training methods and architectures, more evident with full fine-tuning.The authors describe increasing model size as a promising way to enhance molecular modeling.Source: §5.3 · p. 8, Figure 6 · p. 8

6 Conclusion and Future Work

The paper introduces MoleculeQA to address the lack of evaluation for factual discrepancies in molecular LLMs. Its evaluation identifies model shortcomings and factors relevant to molecular comprehension, and it proposes model development, benchmark use in training, and broader data sources as future directions.

6 Conclusion and Future Work

  • MoleculeQA organizes molecular descriptions into a taxonomy and derives factual question-answer pairs through human and LLM efforts.The dataset is presented as a benchmark for molecular factual question answering.Source: §6 · p. 8
  • Evaluation on MoleculeQA reveals shortcomings in existing models and identifies factors the authors consider important for molecular comprehension.Source: §6 · p. 8
  • Future directions include designing a molecular model, using MoleculeQA to train molecular LLMs, and incorporating additional data sources.Source: §6 · p. 8

Limitations

The authors identify data imbalance, limits on full fine-tuning of large models, and the absence of a purpose-built molecular model as limitations.

Limitations

  • Structure and Source data dominate the dataset, reflecting their prevalence in the source materials.The authors plan to add Property and Application data while expanding topic coverage and diversity and safeguarding against data leakage.Source: Limitations · p. 9
  • Computational constraints prevented full fine-tuning of LLMs with 7B parameters and above, so the authors used adaptation-based fine-tuning.Source: Limitations · p. 9
  • The study did not provide a self-designed molecular model based on its benchmark analysis.The authors propose developing one using high-quality domain datasets and an appropriate multimodal fusion strategy.Source: Limitations · p. 9

Potential Risks

MoleculeQA supports factual assessment but may not represent comprehension across all molecules, and it could be misused to create a veneer of reliability for risky models.

Potential Risks

  • MoleculeQA may not represent a model’s comprehension across all molecules, and its evaluations could be used to furnish a veneer of reliability for models with underlying risks.The authors identify misuse as a risk despite the dataset’s reliable data quality.Source: Potential Risks · p. 9

A.1 Data Sources and License

The authors reviewed the origins and legal permissions of the biomolecular data and textual descriptions used to develop MoleculeQA.

A.1 Data Sources and License

  • The data components include biomolecular data and textual descriptions whose origins and legal permissions were scrutinized for compatibility with the research and subsequent use.The authors state that the sources are cited throughout the paper.Source: §A.1 · p. 12

A.2 Details about Taxonomy

MoleculeQA is organized around a hierarchical taxonomy of molecular factual knowledge, with four aspects and more granular subtopics.

A.2 Details about Taxonomy

  • Figure 7 presents the overall hierarchical structure of the taxonomy underlying MoleculeQA, and Table 9 details subtopics and part of the leaf topics across Structure, Source, Property, and Application.Source: §A.2 · p. 12

A.3 Experimental Setup Details

The appendix describes the evaluated model families and training configurations, the taxonomy, and the prompting procedures used in experiments. It also outlines scaling comparisons and the tasks used to construct MoleculeQA.

A.3.1 Baselines

ASPECT SUB TOPICS LEAF TOPICS

  • The taxonomy groups molecular knowledge into four aspects and organizes subtopics under leaf topics, with the displayed table covering only a subset of leaves.Figure 7 shows the overarching hierarchical structure; examples of subtopics and leaf topics are provided in the appendix.Source: Figure 7 · p. 13, Table 9 · p. 14, ASPECT SUB TOPICS LEAF TOPICS · p. 14

DATA SOURCES LICENSE URL LICENSE NOTE

  • The data-resource appendix documents sources and licenses, including T3DB, whose redistribution for commercial purposes requires explicit permission and acknowledgment.Table 10 is described as covering the data resources and licenses used in data collection.Source: DATA SOURCES LICENSE URL LICENSE NOTE · p. 15, Table 10 · p. 15

A.3.2 Hyper-parameters

  • MolT5, MoMu, T5, and BioT5 were fine-tuned for 100 training-set epochs, selecting the best development-set checkpoint for test evaluation; OPT full fine-tuning ran for 60 epochs.MolCA used the original-paper hyperparameters; its 125M and 1.3B versions used one and two GPUs, respectively, and each was fine-tuned for 100 epochs.Source: §A.3.2 · p. 17
  • For LoRA experiments, the authors used Alpaca-LoRA with 20 training epochs and rank 16, α 16, and dropout 0.05.LoRA was applied to attention and feed-forward modules, with module names adjusted where needed.Source: §A.3.2 · p. 17

A.4 Scaling Law in Detail

  • Scaling comparisons covered accuracy across Structure, Source, Property, and Application, contrasting fully fine-tuned T5-based and OPT models with LoRA-tuned BLOOM, Pythia, GALACTICA, and LLaMA2 models.Figure 8 depicts accuracy changes as parameter scale increases; the supplied text does not state its outcomes.Source: §A.4 · p. 17, Figure 8 · p. 18

A.5 Prompt of Different Tasks for LLM

  • MoleculeQA construction used LLM prompts for corpus classification, topic extraction, answer generation, and semantic-consistency validation.Table 11 provides definitions and task contexts for these prompted tasks.Source: §A.5 · p. 17, Table 11 · p. 18

A.6 Few-Shot Details and Prompt Exhibition

  • Few-shot inference used aspect-specific 10-shot prompts with representative samples from different topics; the appendix illustrates a Source prompt and requires selecting one of four options.The prompt rules request answer letters only, in a specified list format.Source: §A.6 · pp. 17, 19, Table 12 · p. 19
Loading 2403.08192v1…