Source-linked AI summary
ChatT2: An Adaptive Framework for Developing a Large Language Model-Based Agent for Natural Product Domain Research
Yihan Wang, Qiandi Gao, Yihui Zhuang, Liangjun Ge, Heqian Zhang, Jiaquan Huang, Zhiwei Qin
TL;DR
General LLMs have limited understanding of specialized natural-product information, and complex biological data are difficult to convey through text alone. ChatT2 uses a mentor, executor, and evaluator with retrieval and scientific tools; it achieved higher rank scores than its GPT-4o mini baseline and the other compared models.
Problem
General LLMs have limited understanding of natural-product corpora amid complex biology, diverse data, sparse documentation, and retrieval challenges.
Source: §1 · p. 2Method
ChatT2 coordinates mentor, executor, and evaluator roles, using retrieval-augmented generation to combine external database information with LLM capabilities.
Source: §2.1 · pp. 3, 5Results
ChatT2 developed from GPT-4o mini achieved higher rank scores than the GPT-4o mini baseline and surpassed the other comparative models.
Source: §2.2 · p. 8Takeaways & Limitations
Iterative multiturn retrieval provided deeper insight for expanding natural-product research in academic or industrial applications.
Source: §2.2 · p. 11Although effective for type II polyketide tasks, ChatT2’s understanding of other domains needs further validation.
Source: §3 · p. 12
Abstract
from arXiv · showhide
Scientific investigations into microbial natural products (NPs) present significant challenges for novices, largely due to the complexity of microbial systems, biochemical diversity, technical skill requirements, and the demands of bioinformatics and data analysis processes. To address these issues, we introduce ChatT2, a large language model (LLM)-based agent that is specifically tailored to the unique characteristics of bacterial type II polyketides. These polyketides form a structurally distinct and therapeutically important NP family. ChatT2 was developed within an autonomous multiagent framework composed of a mentor, an executor, and an evaluator, each with defined responsibilities. The mentor acts as an intermediary between ChatT2 and the user, utilizing chain-of-thought prompting to refine the intent of the user. Under the guidance of the mentor, the executor synthesizes multimodal information via retrieval-augmented generation techniques and seamlessly integrates bioinformatics and cheminformatics tools. The evaluator ultimately assesses the output of the executor to ensure the richness and accuracy of the retrieved information. Our research highlights how ChatT2, designed with this multiagent framework, addresses the challenges faced by general LLMs in terms of understanding limited, specialized corpora and complex biological information and provides both experts and novices with a valuable tool for exploring various NPs of interest. The ChatT2 webserver can be accessed at https://chatt2.site/#/chat.
1 Introduction
Natural-product research draws on diverse biological and chemical knowledge, while general LLMs struggle with specialized corpora, complex information, and reliable outputs. ChatT2 combines a three-agent framework with a type II polyketide benchmark and outperformed the tested models.
1 Introduction
- Type II polyketides are structurally diverse, bioactive compounds that have contributed to antibiotic and antitumour drug development.Research databases contain natural-product data and associated literature, but accessing them can require SQL expertise and interdisciplinary skills.Source: §1 · p. 2
- General LLMs have limited natural-product understanding amid pathway complexity, diverse data, sparse documentation, and retrieval challenges, and may hallucinate without specialized tools.Fine-tuning also faces demands for updates, labelled data, computational resources, and self-assessment.Source: §1 · p. 2
- Single-agent workflows can become unstable across many steps, while text-only interaction makes it difficult to convey specialized biological information accurately.Natural-product research includes biosynthetic pathways, structural data, and spectral data.Source: §1 · pp. 2–3
- ChatT2 outperformed the tested models and was evaluated on 50 open-ended questions about type II polyketides.The benchmark spans five natural-product research scenarios.Source: §1 · p. 3
- The proposed framework assigns distinct responsibilities to a mentor, executor, and evaluator, and introduces a database schema and evaluation standard for natural-product research.The authors present ChatT2 as a natural-language assistant and use type II polyketides as the benchmark case.Source: §1 · p. 3
2 Results
ChatT2 combines iterative mentor–executor work with evaluator oversight and retrieval from multiple information sources. In assessment, it outperformed GPT-4o mini on NPBENCH and produced broader, richer responses across iterative rounds.
2.1 Overview of ChatT2
- ChatT2 is also presented as a tool for natural-product topic mining, including literature reviews and experimental assistance.The framework is intended to provide accurate, reliable, and traceable responses.Source: §2.1 · p. 4
- The mentor plans and iteratively refines queries with the executor, then synthesizes a comprehensive response and relays evaluator-identified anomalies.The mentor uses chain-of-thought prompting to guide retrieval and may help users refine their queries.Source: §2.1 · pp. 4–5
- The executor combines retrieved context from relational and vector databases, image databases, and executable tools for the mentor’s query.Retrieval includes SQL table searches, semantic matching of literature chunks, BM25 image-title matching, and prepackaged scripts.Source: §2.1 · p. 5
- The evaluator checks for unanswerable queries, filters irrelevant information, and tracks citations during preprocessing, retrieval, and answer generation.It can refuse to answer when retrieval fails or relevant information is unavailable.Source: §2.1 · pp. 5–6
2.2 Assessment of ChatT2
- Across NPBENCH categories, ChatT2’s winning rate was more than four times GPT-4o mini’s.Spearman correlation coefficients for repeated automated evaluations exceeded 0.6 across question types.Source: Table 1 · p. 8, §2.2 · p. 8
- Mentor and executor iterations generated diverse follow-up questions, branching from gene mechanisms toward production optimization and clinical applications.The mentor’s later questions draw on executor results from earlier rounds.Source: §2.2 · p. 9
- With more iterations, ChatT2 referenced more documents and produced responses with greater word richness and length than in a single iteration.The five-round assessment found fewer than five documents referenced on average after one iteration.Source: §2.2 · p. 10
- In an open-ended formicamycin inquiry, ChatT2 gathered background, checked the user’s focus, then refined its response through successive questions.The authors present this Auto-mode interaction as a research-assistant example.Source: §2.2 · pp. 10–11
3 Discussion
ChatT2 combines three agent roles with iterative question generation and response to explore bacterial aromatic polyketides; the authors report better accuracy and reliability than existing models. They also identify limits in database preparation, model dependence, and validation beyond type II polyketides.
3 Discussion
- ChatT2’s three-agent framework uses a question-generation–response–question approach, and the authors report greater accuracy and reliability than existing models.The mentor can solicit experimental results or clarification during execution.Source: §3 · p. 11
- The external knowledge database still requires substantial manual filtering and expansion across natural-product subfields.The authors also note that ChatT2 relies heavily on its underlying LLM for intelligence.Source: §3 · pp. 11–12
- ChatT2’s effectiveness is established for type II polyketide tasks, while its understanding of other domains still needs validation.The authors also suggest that a larger internal knowledge base could shift the framework toward individual reasoning and collaborative discussion.Source: §3 · p. 12
- Future work includes broadening the database, adding specialized agents for complex tasks, and incorporating real-time data to adapt to evolving scientific paradigms.The authors also propose adapting the framework beyond natural products to biotechnology and chemistry.Source: §3 · pp. 12–13
4 Experimental Section
The experimental section describes ChatT2’s agent roles, retrieval resources, evaluator, benchmark, assessment procedures, and message-history management. It also details how the system handles uncertain queries and maintains separate interaction contexts.
4.1 Design of the multiagent framework
- The mentor plans and iteratively gathers information, the executor uses memory, databases, or tools, and the evaluator checks reliability; GPT-4o mini serves as the agents’ brain.The authors cite its 128K context window, faster inference, developer-friendly API, and JSON output capability.Source: §4.1 · p. 13
4.2 Description of the Mentor
- The mentor uses chain-of-thought prompts across five research scenarios to generate sequential follow-up questions and a comprehensive report.Its inputs include the initial question, subsequent conversation, and research approach.Source: §4.2 · p. 13
4.3 Description of the Executor
- The executor combines relational, vector, image, and tool resources, including MIBiG and PubChem data, literature, image retrieval, and sequence-analysis tools.Its stated design principles are information diversity, fault-tolerant retrieval, and maximizing context signal-to-noise.Source: §4.3 · pp. 13–14
- Similarity filtering and adjustable recall shape retrieved context, while fuzzy search and mentor clarification address imprecise queries and language variation.The executor can use a user-adjusted self-controlled recall rate; the evaluator can filter irrelevant context and trigger retrieval or clarification when needed.Source: §4.3 · p. 15
4.4 Description of the Evaluator
- The evaluator assesses whether ChatT2 can answer, removes irrelevant retrieved XML content, and checks whether used or unused chunks are cited correctly.It can prompt a refusal, renewed retrieval, clarification, or answer regeneration when checks indicate a problem.Source: §4.4 · p. 16
4.5 Benchmark dataset
- NPBENCH contains 50 type II polyketide problems across biosynthesis, chemical modification and structural optimization, functions, applications, and classification.The benchmark uses definitive expected answers and includes both single-turn and multiturn assessments.Source: §4.5 · p. 16
4.6 Assessment Tasks
- Single-turn assessment ranks models on five response metrics with repeated shuffled GPT-4 rankings and expert review, while multiturn assessment tracks question diversity and answer richness.The multiturn measures include referenced documents, word richness, and text length; termination may follow document-count convergence or mentor assessment.Source: §4.6 · pp. 17–18
4.7 Multiagent Historical Memory Management
- Three message-history lifecycles separately manage user–mentor, mentor–executor, and evaluator interactions so each agent accesses relevant context.The authors state that independent message streams prevent information overlap.Source: §4.7 · p. 18