Source-linked AI summary
Designing a Robust LLM-Based Evaluation System for Agentic AI in Drug Discovery Through Human Alignment
Emma Granqvist, Rocío Mercado, Samuel Genheden
TL;DR
Open-ended agentic drug-discovery outputs are difficult to evaluate with reference metrics or costly expert review, motivating scalable but human-aligned judges. The paper designs and validates such a framework for ChatInvent, then optimizes it with human examples. Alignment with the human majority vote improves from 0.80 to 0.86, while the held-out evaluation reveals output gaps and uncertain phrasing effects.
Problem
Reference-based metrics miss semantic correctness in open-ended outputs, while expert evaluation is too resource-intensive for rapid, large-scale assessment.
Method
The paper develops a ChatInvent LLM-as-a-Judge framework combining four output-quality dimensions with deterministic Tool Call Correctness and human alignment validation.
Results
Alignment with the human majority vote improves from 0.80 to 0.86 after judge optimization using human-annotated examples.
Takeaways & Limitations
Human alignment validation and domain-specific optimization support more robust and scalable evaluation of agentic drug-discovery systems.
Takeaways & Limitations
The judge’s majority-vote target reduces individual-annotator noise, making comparison with pairwise human-human agreement asymmetric.
Abstract
from arXiv · showhide
Agentic large language model (LLM) systems are reshaping scientific workflows in chemistry and drug discovery, but evaluating their open-ended, tool-augmented outputs remains a fundamental bottleneck. Reference-based metrics such as BLEU and ROUGE fail to capture semantic correctness, while expert human evaluation does not scale to the iteration speed these systems demand. The LLM-as-a-Judge paradigm has emerged as a scalable alternative, but existing drug discovery benchmarks deploy LLM judges without validating their alignment with human experts. In this work, we present an LLM-as-a-Judge evaluation framework for ChatInvent, an agentic drug discovery assistant deployed at AstraZeneca, with four contributions. First, we define four output-quality evaluation dimensions---Completeness, Relevancy, Structural Clarity, and Scope Adherence---alongside deterministic Tool Call Correctness checks. Second, we validate the judge through a human alignment study with five expert annotators, comparing Gemini 3.1 Pro, Claude Opus 4.7, GPT-5, and Llama 3.1 70B as candidate judges. Third, we optimize the best-performing judge using few-shot demonstrations of human-annotated examples, improving alignment with the human majority vote from 0.80 to 0.86. Fourth, applying the optimized judge to 70 held-out questions, we surface concrete limitations and find that informal phrasings do not systematically degrade output quality; if anything, it is helpful to have the LLM rewrite the original question before querying the agent. Our framework provides a reusable template for human-aligned evaluation of agentic systems in scientific domains.
1 Introduction
LLM-based agentic systems combine domain tools with generative models in chemistry and drug discovery, creating evaluation challenges that traditional metrics and expert review do not fully solve. The paper proposes a human-aligned, scalable LLM-as-a-Judge framework for ChatInvent.
- Agentic systems orchestrate specialized chemistry tools and databases to support tasks requiring precise calculations, symbolic reasoning, or current knowledge.Examples include planning experiments and integrating computational chemistry software and scientific infrastructure.
- Traditional reference-based metrics are insufficient for open-ended, context-dependent outputs, while expert annotation is too time-consuming for large-scale evaluation.LLM judges offer scalability but require evaluation of their own reliability, bias, and reproducibility.
- The framework evaluates ChatInvent with an LLM-as-a-Judge designed for reliable, automated, and scalable assessment of an agentic drug discovery system.
- The study analyzes judge design, human alignment, model performance, and agentic output quality across ChatInvent’s functionalities.
- The paper focuses on ChatInvent rather than comparing agents because agent capabilities and scope continuously change.
2 Related Work
Related work identifies a tension between scalable automated evaluation and the need for semantically meaningful, human-aligned assessment of agentic scientific systems. Existing drug discovery benchmarks evaluate agents and outputs, but the paper emphasizes validating the evaluator itself.
- BLEU and ROUGE miss semantic nuance, task diversity, and the open-ended nature of generative outputs, while expert evaluation limits rapid large-batch assessment.
- LLM-as-a-Judge scales output evaluation and can exceed 80% agreement with human evaluators on general tasks.However, such judges exhibit systematic biases including self-preference and format or verbosity bias.
- Prompt optimization and LLM-based evaluation have been combined through frameworks such as DSPy and LLM-as-optimizer approaches.
- SciToolEval and MolBench provide comprehensive evaluation efforts for agentic drug discovery systems using multidimensional assessment and tool-call or task benchmarks.
3 ChatInvent: Agentic Drug Discovery System
ChatInvent is a multi-agent drug discovery assistant integrated into AstraZeneca’s discovery pipeline and designed to support the DMTA cycle. Its supervisor routes requests among specialized sub-agents and available computational tools.
- ChatInvent supports stages of the Design–Make–Test–Analyze cycle in the AstraZeneca discovery pipeline.
- The system uses computational tools including ReInvent, AiZynthFinder, and PrecedentFinder, with its tool set expanding over time.Table 1 summarizes available tools, their functions, and responsible sub-agents.
- A supervisor agent directs user requests to appropriate task-specific Design, Synthesis, Analyzer, and Utility sub-agents.
- Sub-agents make the required tool calls, after which the supervisor agent delivers the final response to the user.
4 Evaluation Framework
The evaluation framework extends tool-level checks with LLM-judged assessment of ChatInvent’s user-facing outputs. It combines predefined quality dimensions, deterministic tool-call comparison, multiple candidate judges, and curated test questions.
- 4 Evaluation Framework: The framework evaluates Tool Call Correctness deterministically alongside Completeness, Relevancy, Structural Clarity, and Scope Adherence using an LLM judge.Scores use predefined labels mapped to 0, 0.5, and 1, with Scope Adherence having an exception to the label scheme.
- 4.1 LLM-as-a-Judge: Figure 1 passes ChatInvent’s output and relevant context to the judge, which produces a score and justification for each evaluation dimension.
- 4.1 LLM-as-a-Judge: The judge receives the user question, agentic output, evaluation context, and expected, accepted, and actual tool calls, then returns dimension scores with justifications.An accepted tool-call sequence may be incomplete while still producing valuable information for the user.
- 4.1 LLM-as-a-Judge: Four candidate judges were compared: Gemini 3.1 Pro, Claude Opus 4.7, GPT-5, and Llama 3.1 70B.The models included both API-based and open-weight systems.
- 4.2 Test Questions: The test set began with 20 manually curated questions covering individual tools and multi-tool workflows across ChatInvent’s expected functions.Four LLM-generated variations of each question plus the neutral reference produced 100 main questions.
- 4 Evaluation Framework: Tool-call correctness compares actual sequences with expected complete matches or accepted partial matches defined for each question.
5 Human Alignment
The study assessed human reliability, intra-LLM consistency, and human–LLM alignment for evaluating ChatInvent, then optimized the selected judge with labeled few-shot examples.
- Human annotation: Thirty questions, including five duplicated items, were annotated by five expert AI-for-chemistry researchers using the evaluation rubrics.The duplicated questions supported estimates of human self-consistency and the full set contained 35 presented questions.
- Human annotator reliability: Human self-consistency averaged 0.82 exact match and 0.69 weighted Cohen’s kappa across duplicated questions.One duplicated annotation was randomly removed before the remaining analyses.
- Human annotator reliability: Inter-human agreement was positive and moderate overall, highest for Completeness and lowest for Scope Adherence.Agreement was assessed with Fleiss’s kappa and mean pairwise weighted Cohen’s kappa across annotators.
- Intra-LLM agreement: Across three runs on 30 questions, LLM agreement was generally high, except for GPT-5 on Scope Adherence.GPT-5 generally showed the lowest inter-model agreement and was especially inconsistent for Scope Adherence.
- Human–LLM alignment: Gemini 3.1 Pro achieved the highest overall human alignment among GPT-5, Gemini 3.1 Pro, Claude Opus 4.7, and Llama 3.1 70B.LLM scores were determined by majority vote across three runs, while human preferences also used majority voting.
- Judge optimization: Few-shot signature optimization improved validation alignment from 0.80 to 0.86, after which the optimized judge evaluated the remaining questions.The LabeledFewShotOptimizer selected demonstrations from 20 annotated training samples, with 10 held out for validation.
6 Experiments & Results
The optimized judge evaluated 70 held-out questions across quality dimensions and tool correctness, revealing strong overall performance but substantially weaker Completeness. Scores varied by workflow complexity, question category, and formality, while informal phrasing did not significantly harm performance.
- Overall evaluation: 90% Tool Call Correctness, 86% Relevancy, 84% Scope Adherence, and 90% Structural Clarity were recorded across 70 questions.Completeness was lower: 43% of outputs were rated Complete and 39% Partially Complete.
- Overall evaluation: Nine of 13 incomplete samples resulted from system errors, while four involved incomplete agent outputs or tool calls.Partially complete outputs commonly omitted requested CSV paths, tables, or molecular descriptors.
- Question type: Completeness and Tool Call Correctness were higher for tool questions than workflow questions, whereas Relevancy, Structural Clarity, and Scope Adherence were higher for workflow questions.Workflow questions were more complex and used more tools, leaving more room for error; the latter dimensions were not negatively impacted by that complexity.
- Question category: Completeness was lowest for Mol2Mol, DesignOnly, Design Synthesis, and DesignPropertySynthesis categories, which all used Mol2Mol.Mol2Mol commonly failed to provide the requested CSV path, contributing to lower category scores.
- Question category: AiZynthFinder and Scoring questions had the lowest Scope Adherence and Relevancy because of tool-execution errors.A DesignPropertySynthesis question also received low Scope Adherence after Mol2Mol failed to generate molecules meeting the requested criteria.
- Question formality: The most formal questions produced the lowest scores, while neutral or informal phrasings showed slight score increases and did not significantly harm performance.The reference question also scored below the more informal variants, although standard deviations were high.
7 Discussion
The study demonstrates a human-aligned LLM judge for ChatInvent and uses it to identify output deficiencies, prompt effects, and production constraints. Results support scalable evaluation, but remain bounded by uncertainty and the agent’s changing capabilities.
- Five annotators evaluated ChatInvent outputs alongside the LLM judge, enabling alignment assessment and subsequent optimization toward human preferences.
- The judge identified recurring missing CSV paths and requested properties through lower Completeness scores, highlighting concrete areas for agent improvement.
- Complexity affected Tool Call Correctness and Completeness more than Relevancy and Structural Clarity, while partial answers could remain interpretable.
- Manually formulated questions appeared to yield lower-quality outputs, suggesting that LLM rewriting before agent queries may be advantageous.
- Informal phrasing did not clearly worsen output quality; overly formal phrasing appeared slightly lower-scoring, but the results had high uncertainty.
- Production use may require smoke tests because running the full evaluation or embedding the judge in the agentic loop could be prohibitively expensive.
- Future evaluations should include ambiguously phrased, out-of-scope, erroneous, and ethically questionable questions as ChatInvent’s capabilities expand.
8 Conclusion
The paper concludes that careful LLM-as-a-Judge design is necessary for reliable, scalable evaluation of agentic drug-discovery systems. Human-alignment validation and domain-specific optimization are presented as overlooked design dimensions supporting more robust evaluation.
- Careful LLM-as-a-Judge design is necessary for reliable, scalable, and autonomous evaluation of agentic drug-discovery systems.
- Human-alignment validation and domain-specific judge optimization extend evaluation systems beyond previously overlooked design dimensions.
A Question Formality Levels
The study varies the formality of questions while preserving their underlying tasks. It compares manually written references with LLM-generated rephrasings from highly formal to informal or slang language.
- The same underlying task is rephrased across five formality levels, from highly formal to informal or slang language.
- Variation 0 is the manually written reference question, while variations 1–4 are LLM-generated rephrasings.
- Examples span Synonyms2Smiles, DesignOnly, SynthesisOnly, and DesignSynthesis categories, with molecule and SMILES placeholders filled during evaluation.
B Evaluation Dimension Correlations
Evaluation dimensions are correlated across questions, with Scope Adherence and Relevancy showing the strongest relationship. The authors associate this pattern with nine questions that produced errors and low scores on both dimensions.
- Scope Adherence and Relevancy are highly correlated according to Pearson correlation analysis across evaluation dimensions.
- The other evaluation dimensions are moderately correlated with one another.
- Nine questions produced errors and scored low in both Scope Adherence and Relevancy, likely contributing to their high correlation.