PaperlayerView exact revision on arXiv

Source-linked AI summary

BioinspiredLLM: Conversational Large Language Model for the Mechanics of Biological and Bio-inspired Materials

Rachel K. Luu, Markus J. Buehler

arXiv:2309.08788v2cond-mat.mtrl-scicond-mat.dis-nncond-mat.softcs.LGnlin.AO

TL;DR

Biological-materials knowledge remains largely untranslated into engineering-scale solutions, despite its potential to inform sustainable materials. BioinspiredLLM specializes an open-source conversational model on more than a thousand structural biological-materials articles, and the paper reports accurate recall and promising hypothesis generation. The authors position it as a tool for research assistance and connecting knowledge domains.

  • Problem

    Much knowledge in biological materials remains characterization-focused, and many studied subjects have not reached bio-inspired engineering-scale applications.

    Source: §1 · p. 2

  • Method

    The authors fine-tuned an open-source conversational model on a corpus of over a thousand structural biological-materials articles, using question-answer processing to distill key insights.

    Source: §2.1 · pp. 2–3

  • Results

    BioinspiredLLM reportedly outperforms its base model, recalls biological-materials information, and generated a jackfruit-thorn hypothesis strongly supported by findings from work outside its training dataset.

    Source: §2.4 · p. 6, §3 · p. 8

  • Takeaways & Limitations

    The authors suggest BioinspiredLLM can assist researchers with organized material-property information and may help connect knowledge domains in bio-inspired materials research.

    Source: §2.5 · p. 7, §3 · p. 8

    This version’s knowledge base is limited to the over one thousand articles included in its dataset, whose creation is detailed in the Methods.

    Source: §3.1 · p. 9

Abstract

from arXiv · show

The study of biological materials and bio-inspired materials science is well established; however, surprisingly little knowledge has been systematically translated to engineering solutions. To accelerate discovery and guide insights, an open-source autoregressive transformer large language model (LLM), BioinspiredLLM, is reported. The model was finetuned with a corpus of over a thousand peer-reviewed articles in the field of structural biological and bio-inspired materials and can be prompted to recall information, assist with research tasks, and function as an engine for creativity. The model has proven that it is able to accurately recall information about biological materials and is further enhanced with enhanced reasoning ability, as well as with retrieval-augmented generation to incorporate new data during generation that can also help to traceback sources, update the knowledge base, and connect knowledge domains. BioinspiredLLM also has been shown to develop sound hypotheses regarding biological materials design and remarkably so for materials that have never been explicitly studied before. Lastly, the model showed impressive promise in collaborating with other generative artificial intelligence models in a workflow that can reshape the traditional materials design process. This collaborative generative artificial intelligence method can stimulate and enhance bio-inspired materials design workflows. Biological materials are at a critical intersection of multiple scientific fields and models like BioinspiredLLM help to connect knowledge domains.

1. Introduction

Biological materials offer hierarchical structure–property relationships and potential blueprints for sustainable, high-performance materials, but much research remains characterization-focused and many subjects have not reached engineering-scale applications. Conversational LLMs may help synthesize this literature and connect knowledge domains for bio-inspired materials research.

1. Introduction

  • Despite extensive study, much biological-materials research remains characterization-focused, and many subjects have not reached bio-inspired engineering-scale applications.The field’s hierarchical structure–property relationships offer blueprints for sustainable, high-performance materials.Source: §1 · p. 2
  • Autoregressive conversational LLMs offer a way to specialize scientific models for dialogue, unlike models focused on classification, entity recognition, sequence generation, or property prediction.Prior specialized models include BatteryBERT, OpticalBERT, ProGen, and TransPolymer.Source: §1 · p. 2
  • Biological-materials mechanics is suited to LLM study because its literature is extensive, its knowledge spans domains, and its materials lack standardized written representations.The authors frame this specialization as a step toward accelerating research and discovery in bio-inspired materials.Source: §1 · p. 2

2. Results and Discussion

BioinspiredLLM is built by fine-tuning an open-source model on structural biological-materials literature, with question-answer processing selected for more conversational responses. The section reports stronger knowledge recall with prompting and retrieval, hypotheses about unstudied materials, and workflows that support research and design tasks.

2.1. Model Training and Evaluation Strategy

  • Question-answer processing was selected because it produced more conversational responses with fewer undesirable remnants than the default text approach, despite no large content discrepancy.The final corpus contained over a thousand structural biological-materials articles; the Orca-2-13b fine-tuned model was named BioinspiredLLM.Source: §2.1 · pp. 2–3

2.2. Knowledge Recall

  • On the biological materials exam, prompting BioinspiredLLM with “Think step by step:” raised its baseline score from 82% to about 90%; with RAG, it claimed the top score among all examination categories.RAG also supported numerical knowledge recall.Source: §2.2 · p. 4
  • BioinspiredLLM answered a hoof-mechanics query concisely, linking hydration with pliability and dryness with brittle fracture, while the base-model responses were less focused.The reported comparison includes traditional web search and responses from Llama 13b-chat and Orca 13b.Source: §2.2 · p. 5

2.4. Hypothesis Generation

  • BioinspiredLLM proposed experiments for eucalyptus gumnuts, drawing on characterization approaches from its fine-tuning corpus for a subject the paper says had not been explicitly studied.Suggestions included microscopy, mechanical testing, fracture analysis, water absorption, and biodegradation; the authors connect several suggestions to plant-material characterization articles.Source: §2.4 · p. 6
  • For jackfruit, BioinspiredLLM predicted that thorns absorb impact energy and control crack propagation, matching findings in a study outside its training data.The paper reports that the model had not seen that study; the predicted crack path followed around thorns along the foamy matrix.Source: §2.4 · p. 6

2.5. Assistive Tasks

  • For a requested biological-materials dataset, BioinspiredLLM returned an organized list pairing materials with structures and mechanical properties.Examples include bone with cortical and cancellous tissues and compressive strength, and nacre with a layered structure and stiffness.Source: §2.5 · p. 7
  • BioinspiredLLM proposed combinations of biological structures with property rationales and behavioral hypotheses, then used selected ideas to generate images and 3D models.The described workflow uses Stable Diffusion 2.0 for images and converts a selected image into a 3D model.Source: §2.5 · pp. 7–8

3. Conclusions

BioinspiredLLM is presented as a fine-tuned conversational model for structural biological materials mechanics, with conclusions emphasizing recall, research support, and collaboration. The authors also stress output verification and corpus limits, while identifying dataset, training, and multimodal improvements for future work.

3. Conclusions

  • BioinspiredLLM is presented as a specialized conversational model that can help accelerate research in structural biological materials mechanics.The conclusions describe it as an expert in this field.Source: §3 · p. 8
  • The authors report accurate recall, creative insight generation, dataset support, and promise in workflows collaborating with other generative AI models.They state that recall is especially effective with retrieval augmentation and that collaborative workflows could decrease design time and resources.Source: §3 · p. 8

3.1. Limitations and Disclaimers

  • Researchers should verify model responses because errors remain possible, and should recognize the limits of the corpus.The authors also recommend tracing informational responses to source authors and disclosing the model's contribution to generative ideas.Source: §3.1 · pp. 8–9

3.2. Outlook and future work

  • The authors frame BioinspiredLLM as one tool in a broader scientific toolkit, whose usefulness depends on the question and validation setting.They describe LLMs as able to apply recall, synthesis, and translation across domains.Source: §3.2 · p. 9
  • Future work could assess prediction confidence, develop active learning, or scale the model, although larger models require substantially more hardware.The authors also mention larger and better-curated datasets, human-derived question-answer pairs, and training that includes equations, tables, figures, and plots.Source: §3.2 · p. 9
  • Only 387 of the 1,034 training articles had some form of open-access designation, making non-open-access articles harder to obtain through text and data mining.The authors discuss combining corpora from different scientific disciplines as another future direction.Source: §3.2 · pp. 9–10

Large Language Model Selection and Training

The authors fine-tuned Llama-2-13b-chat and Orca-2-13b for BioinspiredLLM, selecting the Orca-based model as the final model after observing a performance increase. They used LoRA and 4-bit quantization, with next-token likelihood as the training objective.

Model Selection

  • Orca-2-13b was selected as the final BioinspiredLLM model after the authors observed a noticeable performance increase over the Llama-2-13b-chat framework.Orca 2 uses Llama 2 as its base and is fine-tuned for enhanced reasoning.Source: Large Language Model Selection and… · p. 10

Fine-tuning

  • LoRA fine-tuning freezes pretrained weights and biases while adding smaller trainable layers, aiming to learn new tasks while retaining original knowledge.The authors used rank r = 96 with α = 16 and 4-bit quantization during training and inference.Source: Large Language Model Selection and… · p. 10

Implementation

  • The implementation used a 40-layer transformer with rotary positional embeddings, and training ran for approximately 3,000 steps.The model was developed in PyTorch and implemented in the Hugging Face ecosystem.Source: Large Language Model Selection and… · p. 10

Training Objective

  • Training maximized the likelihood of predicting each next token from preceding text, including conversational context and the system prompt.The target at each position was the actual next token.Source: Large Language Model Selection and… · p. 10

Dataset Generation

The authors assembled a corpus focused on biological-material mechanics, used it to create training data, and evaluated recall with an author-designed question set. They also describe retrieval augmentation and a chat interface as parts of the system.

Dataset Generation

  • The Web of Science search for “biological materials mechanical hierarchical structure” returned 1,056 English results, of which 1,034 articles were obtained for training.The search was conducted on 31 July 2023; the reported yield was 98%.Source: Dataset Generation · pp. 10–11

Model Evaluation

  • The authors evaluated knowledge recall with a 100-question multiple-choice examination because no established benchmark existed for this domain.Questions covered general, specific, non-biological, and numerical topics; the authors caution that the examination may not cover the field's full breadth.Source: Model Evaluation · p. 11

Retrieval-Augmented Generation (RAG)

  • Retrieval-augmented generation searches a database for query-relevant information and supplies it as additional context to the language model.The authors built a training-set vector database and divided corpus texts into 10,463 chunks for RAG experiments.Source: Retrieval-Augmented Generation (RAG) · pp. 11–12

Interactive Gradio Chat Application

  • The Gradio chat interface uses streaming output and could be extended with literature search or context from papers and other documents.The authors identify physics simulations and image synthesis as possible future extensions, not functions used in the reported experiments.Source: Interactive Gradio Chat Application · p. 12

Code Availability

The authors provide a repository for the model, code, trained weights, and data, and identify supporting-information files containing corpus and assessment materials.

Code Availability

  • The model, code, trained weights, and data are available from the BioinspiredLLM GitHub repository.The passage gives the repository address.Source: Code Availability · p. 12

Supporting Information

  • Supporting information includes CSV files for the 1,034-article corpus, processed question-answer pairs, and knowledge-recall questions and answers..Source: Code Availability · p. 12

Acknowledgements and Conflict of Interest

  • The paper acknowledges its funding sources and declares no conflict of interest.Funding includes NSF, Army Research Office, ONR, USDA, MIT's Generative AI initiative, and the IBM-MIT Watson Center.Source: Code Availability · p. 12

Related papers

Loading 2309.08788v2…